Integration Request: Support for Portkey as a Remote Language Model Provider

[kaushikb11]

We are currently running multiple experiments with the DSPy framework, which is designed to facilitate seamless interaction with various language models. Given the impressive capabilities and the wide range of models supported by Portkey, we are interested in integrating Portkey as a remote language model provider within the DSPy framework.

Portkey offers a unified API that simplifies the process of interacting with various language models, including those from Azure OpenAI, OpenAI, Anthropic, and more.

This integration would not only expand the framework's versatility but also provide the users the option to leverage the AI Gateway with Load Balancing and Rate Limiting support.

We have reviewed the Portkey documentation and believe that the integration process would be straightforward, especially with the Python SDK provided by Portkey.

We are happy to work and contribute with this integration. Do let us know if this sounds good and we can start with the PR.

[roh26it]

This sounds great @kaushikb11 - We'd be happy to work on this with you as well.

I was talking to @CShorten about this and he asked me to look at the cohere.py implementation in https://github.com/stanfordnlp/dspy/tree/main/dsp/modules

[CShorten]

Hey @roh26it, maybe one thing to note on this is that the Cohere chat API no longer supports returning multiple generations per call so we rewrote a few things around how DSPy then parses the response. But I think hopefully still gives a good sense of how to integrate portkey -- if you don't mind sketching out the initial PR draft, I can take a look from there. I guess the key distinction maybe is how you interface the model -- would is be a list[str] of models with portkey? And then re the above discussion, each one would need to either need to return a single response or multiple responses consistently.

[roh26it]

Yes, with Portkey we'd have a configuration shared that could call one or more models but for the sake of this integration - would only be 1 model.

Portkey would add the flexibility to use any of the models and providers without needing to integrate with all of them.

[isaacbmiller]

Hey @roh26it, we are currently doing a backend refactor inside of #424.

Currently, we are integrating with LiteLLM as the default provider for many models using the OpenAI syntax. I would hold off on this project until after #424 is merged in, as DSPy will better handle integrating providers like this.

[roh26it]

@isaacbmiller curious if you'd like to keep it open to using any OpenAI compatible proxy?

[isaacbmiller]

Yup anything that conforms to BaseLM.

Here is what litellm looks like. https://github.com/stanfordnlp/dspy/blob/ed8bc1cbc756522d32b445739352896011048506/dspy/backends/lm/litellm.py

Things are subject to change with BaseLM, so I wouldn't work on an implementation until this is merged.