In Ray summit, @pcmoritz talked about the embedding models (especially the GTE-base) in the session Developing and Serving RAG-Based LLM Applications in Production. It would be great if we could also have a model config in models/continuous_batching for this CPU model so the developer can host all the models relevant to the RAG in ray-llm.
In Ray summit, @pcmoritz talked about the embedding models (especially the GTE-base) in the session
Developing and Serving RAG-Based LLM Applications in Production
. It would be great if we could also have a model config inmodels/continuous_batching
for this CPU model so the developer can host all the models relevant to the RAG in ray-llm.