Support other LLM providers

[jkomoros]

• [ ] Create example eseds to try the same sub-graphs with different providers and see which ones are better
• [x] Document
• [ ] countTokens should be able to take multiple things at once for one round-trip to production API for the ones that are not locally computed. (assuming those apis allow passing multiple bits of text)
• [ ] Cache countTokens results for strings (since for some providers they go to the server)
• [x] Move growPrompt switch statement to being in providers/index.ts
• [x] computePrompt et al, if verbose is true, should print the model being used to the console.
• [ ] Pop out the code to create a mocked embedding and use environment.random()
• [x] getProvidersWithAPIKeys() should have a defined priority order (update documenation in Selecting which model to use ) section
• [ ] Suppress node warning when running using Google (fetch warning)
• [ ] modelName should be a verified set of literals and each provider should verify it matches what they expect
• [ ] google tokenCount should return a mocked count if environment says mock
• [x] if recall embedding is '' then just use a random one
• [x] Move countTokens switch statement to being in providers/index.ts
• [x] Move providers/index.ts to llms.ts?
• [ ] Factor out Google request library and fetches into one google provider (and maybe put all of openai provider into one file too)
• [ ] if you recall from a memory that was set with a different embedding size it fails (maybe not?)
• [x] have a way to switch LLM providers all at once, (completion_model and embedding_model at once) (if provider is set and embedding_model or completion_model is not, use the default model of type for provider. And then set provider based on the first provider that has a non-changeme API key)
• [x] Some way for models to detect which providers have API keys so they can automatically iterate through models to test sub-grapsh (this might just be a meta-node, if getAPIKey were changed to by default not throw but return '')

[ishaan-jaff]

Hi @jkomoros I believe we can help with this issue. I’m the maintainer of LiteLLM https://github.com/BerriAI/litellm

TLDR: We allow you to use any LLM as a drop in replacement for gpt-3.5-turbo. You can use our proxy server or spin up your own proxy server using LiteLLM

Usage

This calls the provider API directly

from litellm import completion
import os
## set ENV variables 
os.environ["OPENAI_API_KEY"] = "your-key" # 
messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# falcon call
response = completion(model="falcon-40b", messages=messages)