search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
13.8k stars 1.62k forks source link

Make oidc call async #3858

Open krrishdholakia opened 5 months ago

krrishdholakia commented 5 months ago

https://github.com/BerriAI/litellm/blob/a6a84e57cece367f5ba9d1ebccd42d59649b6a31/litellm/utils.py#L10040

We cannot have the oidc calls be sync calls. this would be a blocking operation for async calls.

Manouchehri commented 5 months ago

Could we merge in the OIDC PRs before I tackle this? :)

krrishdholakia commented 5 months ago

Sounds good - which ones are ready for review?

Manouchehri commented 5 months ago

3687 and #3712

krrishdholakia commented 5 months ago

both are blocked on this issue, since both make the same sync call @Manouchehri

Manouchehri commented 5 months ago

Changing def get_secret to async def get_secret is a fair amount of changes. Can you confirm that's what you want me to do before I actually do it?

Manouchehri commented 5 months ago

I'm still kinda confused on the reasoning behind why this should be async. Having blocking on the first auth request makes sense to me, since otherwise multiple calls in parallel will unnecessarily make multiple OIDC tokens. I don't see how this would improve perf, since requesting multiple OIDC tokens will not be faster than just waiting for the first one to return?