Consider the case where the dev wants to connect to Azure OpenAI.
How Azure OpenAI works is, they'll provision a limited number of dedicated LLMs for you. These LLMs need to be queried in different styles such as round-robin, or based on lowest rate limit. In such a case, the connection pool should take a list of Azure OpenAI endpoints and credentials.
This should be possible with custom adapters now. i don't think we should implement this ourselves, it's a niche case that maybe someone in the community will be able to share code for.
Consider the case where the dev wants to connect to Azure OpenAI.
How Azure OpenAI works is, they'll provision a limited number of dedicated LLMs for you. These LLMs need to be queried in different styles such as round-robin, or based on lowest rate limit. In such a case, the connection pool should take a list of Azure OpenAI endpoints and credentials.
For example:
we need more requirements, but some considerations & questions i'm expecting are: