search

deepset-ai / haystack

AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.72k stars 1.92k forks source link

Azure / OpenAI rate limit headers #8164

Open dmvieira opened 3 months ago

dmvieira commented 3 months ago

Is your feature request related to a problem? Please describe. Now OpenAIGenerator and OpenAIChatGenerator only returns choices without the response HTTP headers with rate limit information.

Describe the solution you'd like Receive from OpenAIGenerator and OpenAIChatGenerator response headers or rate limit information.

Describe alternatives you've considered Using OpenAI lib without Haystack

vblagoje commented 2 months ago

I think we can easily add this data to meta dict of the ChatMessage, wdyt @dmvieira ? I doubt we'll make any changes to OpenAIGenerator as they are slowly becoming obsolete.

dmvieira commented 2 months ago

Great! I don't know if it's a pattern only for OpenAI, but adding requests headers for all generators perhaps make sense. What do you think?

dmvieira commented 2 months ago

Looking the code, headers will appear in every reply message and it's for the hole request. Idk if it's good...

Captura de Tela 2024-09-04 às 13 11 33