Set up different LLM providers for smart, fast and embedding LLMs

assafelovic / gpt-researcher

LLM based autonomous agent that does online comprehensive research on any given topic

https://gptr.dev

Apache License 2.0

14.25k stars 1.86k forks source link

Set up different LLM providers for smart, fast and embedding LLMs #539

Closed vanetreg closed 3 months ago

vanetreg commented 4 months ago

I think it makes sense to use: gpt-4 as smart LLM, some Groq / Openrouter / Huggingface / Ollama LLM as fast and eg. Nomic Embed (via GPT4All) as embed model, but now we are not able to set fast and smart LLM providers separatedly up AFAIK ( checking config.py ), at least not without Langchain, so pls. consider my idea :)

wofeishenling commented 4 months ago

yeah, i also want it

assafelovic commented 4 months ago

Hey, big release tomorrow that will include this :)

vanetreg commented 4 months ago

Hey, big release tomorrow that will include this :)

As far as I see:

we are still not able to set up different LLM providers for smart (eg. OpenAI) and fast LLMs (eg. Groq, Ollama, HF of Openrouter free)
Openrouter config options are not in docs. It would be a good option with their FREE models as fast LLM to play around with GPT Researcher.

Pls. note: https://docs.gptr.dev/gpt-researcher/config a link from here is not found !

xenstar commented 4 months ago

Yes, please add openrouter as we can integrate with different LLMS easily and see which one is giving better output :)

Speedway1 commented 3 months ago

For locally hosted LLMs this feature is essential, because it allows for different machines to provide each type of LLM which enables for more GPUs to be used preloaded with the LLms, which is much faster. Has one got this working?