search

distantmagic / paddler

Stateful load balancer custom-tailored for llama.cpp
MIT License
518 stars 22 forks source link

(stastsd) not wokring idle slot with latest llama.cpp #19

Open bodybreaker opened 2 weeks ago

bodybreaker commented 2 weeks ago

paddler_slots_idle not working with lastest llama.cpp

us58 commented 1 week ago

The slot states were moved from the /health endpoint to a new /slots endpoint. See https://github.com/ggerganov/llama.cpp/pull/9056.

mcharytoniuk commented 1 week ago

@us58 Thanks for letting me know. I am working on the update now.

I also mentioned the issue in llama.cpp discussions, such breaking change was kind of unexpected: https://github.com/ggerganov/llama.cpp/discussions/9276

mcharytoniuk commented 1 week ago

@bodybreaker @us58 I released a new version that uses the new /slots endpoint from llama.cpp server: https://github.com/distantmagic/paddler/releases/tag/v0.7.0

Please let me know if that solved StatsD issue.