search

llmware-ai / llmware

Unified framework for building enterprise RAG pipelines with small, specialized models
https://llmware-ai.github.io/llmware/
Apache License 2.0
4.26k stars 808 forks source link

Could you please showcase one end-to-end example to run a model over Kubernetes cluster (AKS, etc) with one RAG implementation please? #206

Open ajmal-yazdani opened 7 months ago

ajmal-yazdani commented 7 months ago

I can download the model locally and can write a simple chat application.

But question is how we can run this model over Kubernetes cluster and run the RAG application.

Could you please guide with some sample?

doberst commented 7 months ago

Great feedback - and definitely something on our near-term roadmap. Will come back to you soon. Also, let us know if you would be interested potentially to collaborate with us on building up the Kubernetes script around this use case.

chair300 commented 7 months ago

@ajmal-yazdani This is a great suggestion. I have been working on a websocket chat agent already. While its not ready for checkin and/or demo yet it will be coming soon.

The secondary point about sample kubernetes code to deploy and run RAG is very interesting. Some of the kubernetes code is dependent of the specific use case and model since the model is embedded within the application. I am happy to put together very simple kubernetes deployment code bases on docker containers (I need to build the containers first).

I hope to have the kubernetes code by mid January, with the docker containers built and available before then.

ajmal-yazdani commented 7 months ago

Great feedback - and definitely something on our near-term roadmap. Will come back to you soon. Also, let us know if you would be interested potentially to collaborate with us on building up the Kubernetes script around this use case.

Something for sure I can try on my AKS cluster or local docker.

ajmal-yazdani commented 7 months ago

Thank you very much!

chair300 commented 5 months ago

I just updated the docker image which can run llmware examples in a aks cluster or local cluster. There is a docker-compose file which provides the extra database infrastructure. You will still need to setup the env's but its a start.

chair300 commented 4 months ago

I have a pull request outstanding which will allow you to play with a full docker-compose full service infrastructure. Please see my pull request for the update.

chair300 commented 4 months ago

with docker-compose from the devcontainer folder you can run the following command and then all the infrastructure will be up. Docker exec -it into the llmware container and run the examples. Its all set and easy. Please let me know if you have any further questions.