search

MeetKai / functionary

Chat language model that can use tools and interpret the results
MIT License
1.4k stars 108 forks source link

Dockerfile and official docker image? #276

Open nvonpentz opened 1 week ago

nvonpentz commented 1 week ago

Hi, I'd like to run functionary via Kubernetes, so I'm looking for a dockerfile to build the image or an official docker image to use. Is that something that you would be interested in supporting?

The context is that I am currently running functionary by passing meetkai/functionary-small as the --model param for the official vLLM docker image https://hub.docker.com/r/vllm/vllm-openai/. However, the official vLLM image doesn't support tool_choice=auto for the chat API, which I see is supported in this repo. So I'm looking to switch to this repo to get that functionality.

jeffreymeetkai commented 1 day ago

Hi @nvonpentz, apologies for the really delayed response. I was tied down with other feature work and have not had much time to focus on this.

I think this suggestion is good and I will review your PR. It complements with one of the features which I'm working on right now that it is deployment through SkyPilot. We can build and push your images to dockerhub and deploy to cloud or kubernetes cluster manually or via SkyPilot.

Actually, I am also working on a new server powered by SGLang in #277 . If you are interested, would you like to include the dockerfile for that in your PR? There are base images available by LMSys here