GroqCloud support

[torepettersen]

Hi, thanks for creating this nice library 😊

I recently came accros GroqCloud. It seems like an alternative to OpenAI and is powered by open source moduls like Mixtral, Llama3 and Gemma. They have two big advantegaes over OpenAI which is speed and price.

So just qurious if it would be possible and makes sense to add support for GroqCloud?

[brainlid]

Does it use an OpenAI compatible API? If so, it may be supported right now. I haven’t looked into at this point and would love any insights you have on that.

On Tue, Jun 11, 2024 at 11:16 AM Tore Pettersen @.***> wrote:

Hi, thanks for creating this nice library 😊

I recently came accros GroqCloud https://groq.com/. It seems like an alternative to OpenAI and is powered by open source moduls like Mixtral, Llama3 and Gemma. They have two big advantegaes over OpenAI which is speed and price.

So just qurious if it would be possible and makes sense to add support for GroqCloud?

— Reply to this email directly, view it on GitHub https://github.com/brainlid/langchain/issues/136, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGFQGEZST7OXS6VBFL2NVTZG45JFAVCNFSM6AAAAABJE3IA36VHI2DSMVQWIX3LMV43ASLTON2WKOZSGM2DOMBRHA4TQNY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[yvrhlp]

A few nuances according to Groq docs: https://console.groq.com/docs/openai

OpenAI Compatibility Groq's APIs are designed to be compatible with OpenAI's, with the goal of making it easy to leverage Groq in applications you may have already built. However, there are some nuanced differences where support is not yet available.

The following fields are not supported and will result in a 400 error if they are supplied:

logprobs logit_bias top_logprobs

[brainlid]

@yvrhlp thanks! We're not using any of those. The ChatOpenAI module supports overriding the endpoint so that might be working right now.

Please let me know if you try it. There are a number of services like Groq that creating a "supported" matrix could be really helpful for people.

[torepettersen]

Thanks, I made a basic test and seems to work.

iex> llm = LangChain.ChatModels.ChatOpenAI.new!(%{endpoint: "https://api.groq.com/openai/v1/chat/completions", model: "llama3-70b-8192", api_key: key})
iex> LLMChain.new!(%{llm: llm}) |> LLMChain.add_message(Message.new_user!("Hello!")) |> LLMChain.run()
%LangChain.Message{
   content: "Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?",
   index: 0,
   status: :complete,
   role: :assistant,
   name: nil,
   tool_calls: [],
   tool_results: nil
 }

[brainlid]

Awesome! If you keep playing with it, it would be great to hear how it goes. Thanks for reporting back.

On Thu, Jun 13, 2024 at 2:19 PM Tore Pettersen @.***> wrote:

Thanks, I made a basic test and seems to work.

iex> llm = LangChain.ChatModels.ChatOpenAI.new!(%{endpoint: "https://api.groq.com/openai/v1/chat/completions", model: "llama3-70b-8192", api_key: key})iex> LLMChain.new!(%{llm: llm}) |> LLMChain.add_message(Message.new_user!("Hello!")) |> LLMChain.run()%LangChain.Message{ content: "Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?", index: 0, status: :complete, role: :assistant, name: nil, tool_calls: [], tool_results: nil }

— Reply to this email directly, view it on GitHub https://github.com/brainlid/langchain/issues/136#issuecomment-2166813906, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGFQGEC52VNPLXK3MFQN2DZHIEH7AVCNFSM6AAAAABJE3IA36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWHAYTGOJQGY . You are receiving this because you commented.Message ID: @.***>