Open nvonpentz opened 1 week ago
Hi @nvonpentz, apologies for the really delayed response. I was tied down with other feature work and have not had much time to focus on this.
I think this suggestion is good and I will review your PR. It complements with one of the features which I'm working on right now that it is deployment through SkyPilot. We can build and push your images to dockerhub and deploy to cloud or kubernetes cluster manually or via SkyPilot.
Actually, I am also working on a new server powered by SGLang in #277 . If you are interested, would you like to include the dockerfile for that in your PR? There are base images available by LMSys here
Hi, I'd like to run functionary via Kubernetes, so I'm looking for a dockerfile to build the image or an official docker image to use. Is that something that you would be interested in supporting?
The context is that I am currently running functionary by passing
meetkai/functionary-small
as the --model param for the official vLLM docker image https://hub.docker.com/r/vllm/vllm-openai/. However, the official vLLM image doesn't support tool_choice=auto for the chat API, which I see is supported in this repo. So I'm looking to switch to this repo to get that functionality.