Adding a support for Cohere Command-R, Anthropic Claude, and Gemini APIs

[adam2am]

Hey, I would like to suggest support for integrating additional language model APIs beyond just OpenAI. Specifically, it would be very helpful to have the ability to use:

• Cohere API (including recent Command-R model with amazing retrieval-augmented generation)
• Anthropic API (Claude model)
• Google Gemini API
• Ollama Local LLMs (for the sake of those, who can't share the code cuz of the NDA)

These models rank among the top 10 AI language models according to benchmarks like https://chat.lmsys.org/ and provide capabilities complementary to OpenAI's models.

The recent Command-R model from Cohere is particularly compelling for its strong retrieval-augmented capabilities using its embeddings. And the Claude model from Anthropic has received acclaim for its coherence and abilities to code.

Having this flexibility would be incredibly valuable. Would be amazing if you consider adding it!

[IvoP1]

It would be good to add AWS Bedrock API to that list. It supports Claude, LLama, etc...

[andr0s]

Just add litellm support. Everything already included there. No need to add support for each and every LLM and provider

[yuntongzhang]

We are currently working on the LiteLLM support. It should be ready sometime next week.

[stevensu1977]

Just add litellm support. Everything already included there. No need to add support for each and every LLM and provider

LiteLLM is a great project, but some models have no native support for function calling , etc. Amazon Bedrock Claude3

[andr0s]

@stevensu1977 well, the whole point is to have the ability to run open source and/or free things. Or experiment with different LLMS. For example, I'm really curios about Llama 3 70b and that big llm from databrix - in my tests on other projects, they showed pretty decent results, despite being dirt cheap or even free (depending on where you run them). I think others also willing to experiment or reduce costs, otherwise people would just be happy with ChatGPT and Claude 3, and this ticket wouldn't even exist. Another problem with close source models that I noticed - sometimes they change things and it just breaks everything. I'm using Cursor IDE with ChatGPT and the same model suddenly started to perform much worse, basically breaking the whole code, removing tons of my code and adding stubs like "... unchanged code... " which was extremely annoying. I think it wouldn't happen if the model and the backend layer above it would be exactly the same, which is guaranteed only when using open source solutions

[yuntongzhang]

ACR now supports Claude models through LiteLLM and Llama3 through Ollama (added in https://github.com/nus-apr/auto-code-rover/pull/32). Feel free to try it out :)

[rezzie-rich]

can you please add support for all open LLM through liteLLM or ollama? currently, for code generation, codeQwen 1.5 7b is ranked as high as GPT-4 and it's only a 7b model so it can easily run locally pretty quickly.