[Question]: RAG Inference Latency

[Prathit-tech]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

Hi, I have a RAG system that uses the Zephyr 3b model and uses llama-index for the VectorStoreEmbedding and inference. My problem is with the latency as it seems to take 300s to generate an output. Is there any way that i can get this latency to GPT levels or atleast 60s?

[dosubot[bot]]

Hey there, @Prathit-tech! Great to see you back. Hope you're doing well! 😊

Based on the context provided, it seems that latency issues with the Zephyr 3b model and LlamaIndex have been a common concern among users. However, there are several strategies you can employ to reduce the latency of your RAG system.

1. Batch Processing: If your application's logic allows, batch requests to the Zephyr model and LlamaIndex can significantly reduce latency. Batching reduces the overhead of individual requests and can utilize the model and index more efficiently.

2. Asynchronous Operations: The provided code includes asynchronous methods (async_add, adelete, aquery) that can be leveraged to improve performance. Asynchronous operations allow your system to handle other tasks while waiting for the response from the Zephyr model or LlamaIndex, effectively reducing the perceived latency.

3. Optimize Query and Indexing Parameters: Review and optimize the parameters used for querying and indexing. For example, reducing the similarity_top_k parameter in the query method can decrease the amount of computation required to find the most similar nodes, thus reducing latency.

4. Pre-compute and Cache Embeddings: If possible, pre-compute embeddings for frequently queried texts or nodes and cache them. Accessing pre-computed embeddings from a cache is much faster than computing them on-the-fly.

5. Infrastructure Improvements: Ensure that the infrastructure hosting the Zephyr model and LlamaIndex is optimized for performance. This includes using high-performance computing resources, optimizing network latency, and ensuring that the storage system can quickly read and write data.

6. Parallel Processing: If your application's architecture allows, process multiple requests in parallel. This approach can significantly reduce the overall latency for processing multiple requests.

7. Profile and Optimize Code: Use profiling tools to identify bottlenecks in the integration code between your RAG system and the Zephyr model/LlamaIndex. Optimizing these bottlenecks can lead to significant latency reductions.

8. Review Zephyr and LlamaIndex Documentation: Both Zephyr and LlamaIndex may have specific recommendations or features for reducing latency. Review their documentation for any such advice or features that you might not be utilizing.

You can find more details about these methods in the ZepVectorStore class in the llama_index package.

Unfortunately, I couldn't find specific latency benchmarks for the Zephyr 3b model when used with LlamaIndex for VectorStoreEmbedding and inference. The actual latency can vary based on several factors, including the specific configuration of your system, the complexity of the queries, and the size of the data being processed.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• Reaction is very slow, is there any params to reduce reaction time?
• llama-index-integrations/vector_stores/llama-index-vector-stores-zep/llama_index/vector_stores/zep/base.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

Either get a better GPU (lol) or try using something like TGI or vLLM