[Request] - Adding functionality of other commercial LLMs ?

yash-srivastava19 commented 5 months ago

Apart from the commercial LLMs that are available in the module, are you guys working on introducing other models such as Claude, Bard or similar. If that's something on the pipeline, I would like to contribute.

okhat commented 5 months ago

Hey Yash! We want to add support for LangChain LLMs out of the box, so you can import any of their LLMs and use them in DSPy. We are just starting to look into what’s needed for that

yash-srivastava19 commented 5 months ago

Sure, if there's something where I can be of some utility, hit me up.

ukituki commented 5 months ago

Adding support for more llms, both commercial and open source, can be handled either on case by case basis or in a more scalable way with a good abstraction. I'm not sure about the downstream implications but perhaps the inspiration from llama_index's ServiceContext may be helpful (https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html). Btw, it also supports Langchain llms

It'd be fantastic if we could use their llm abstractions, like that:

Langchain llm:

from langchain.llms import OpenAI
from llama_index.llms import LangChainLLM

model = "gpt-3.5-turbo"
llm_llamaindex = LangChainLLM(llm=OpenAI())
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)

dspy.settings.configure(lm=llm_llamaindex)

Llama2 via TogehterAI:

from llama_index.llms import TogetherLLM

model = "meta-llama/Llama-2-70b-chat-hf"
llm_llamaindex = TogetherLLM(
     model=model, api_key= os.getenv("TOGETHERAI_API_KEY"),
 )
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)
dspy.settings.configure(lm=llm_llamaindex)

yash-srivastava19 commented 5 months ago

Adding support for more llms, both commercial and open source, can be handled either on case by case basis or in a more scalable way with a good abstraction. I'm not sure about the downstream implications but perhaps the inspiration from llama_index's ServiceContext may be helpful (https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html). Btw, it also supports Langchain llms

It'd be fantastic if we could use their llm abstractions, like that:

Langchain llm:
from langchain.llms import OpenAI
from llama_index.llms import LangChainLLM

model = "gpt-3.5-turbo"
llm_llamaindex = LangChainLLM(llm=OpenAI())
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)

dspy.settings.configure(lm=llm_llamaindex)
Llama2 via TogehterAI:
from llama_index.llms import TogetherLLM

model = "meta-llama/Llama-2-70b-chat-hf"
llm_llamaindex = TogetherLLM(
     model=model, api_key= os.getenv("TOGETHERAI_API_KEY"),
 )
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)
dspy.settings.configure(lm=llm_llamaindex)

Good abstractions, as that of Langchain would be really good for scaling. I will read through the ServiceContext and then we can we see how to take it from there. If you have any suggestions, or want to work together on this, we can do that !

stillmatic commented 5 months ago

Many commercial providers work out of the box with the OpenAI library though. For example:

mixtral = dspy.OpenAI(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    max_tokens=250,
    model_type="chat",
    api_key=os.environ["TOGETHER_API_KEY"],
    api_base="https://api.together.xyz/v1",
)

dspy.settings.configure(lm=mixtral)

psykhi commented 5 months ago

Sorry if I'm misunderstanding but does adding support for other LLMs mean supporting chat APIs? (referring to this issue Because at the moment the way prompts are built is very much tailored for instruct models.

okhat commented 5 months ago

Hey @psykhi , I think these two issues are separate. I left some comments in the other issue.

yash-srivastava19 commented 5 months ago

Any PR that is working on this issue?