thesurlydev
commented
4 months ago Taking a quick look at a response from Groq using curl it appears the response object is a little different from the ChatCompletion response. Here's an example response from Groq:
{
"id": "chatcmpl-ee2adcbd-3697-47e9-88b9-d4a5c4ba2032",
"object": "chat.completion",
"created": 1713808705,
"model": "mixtral-8x7b-32768",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Fast language models are important for a variety of reasons, including:\n\n1. Real-time applications: Fast language models can process and generate text in real-time, making them well-suited for applications such as chatbots, virtual assistants, and real-time translation.\n2. Large-scale processing: Fast language models can handle large volumes of text data quickly and efficiently, making them useful for tasks such as indexing and searching large corpora of text.\n3. Low-resource environments: Fast language models can run on devices with limited computational resources, such as smartphones and embedded devices, making natural language processing (NLP) capabilities accessible to a wider range of users and applications.\n4. Interactive exploration: Fast language models can be used to interactively explore and manipulate text data, allowing users to quickly and easily experiment with different text generation prompts and settings.\n5. Cost-effective: Fast language models can be less computationally intensive, which can result in lower costs for training and deployment, as well as reduced energy consumption.\n\nOverall, fast language models are important for enabling a wide range of NLP applications, from real-time chatbots to large-scale text processing, that can run efficiently and effectively in a variety of environments."
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 18,
"prompt_time": 0.006,
"completion_tokens": 267,
"completion_time": 0.472,
"total_tokens": 285,
"total_time": 0.478
},
"system_fingerprint": "fp_7b44c65f25",
"x_groq": {
"id": "req_01hw3fb11dft4t8ny3htzw8ga3"
}
}
Mikl38400
tzolov
Expected Behavior
Spring AI should support Groq with nothing but configuration changes.
Current Behavior
Although Groq's documentation states that it's compatible with OpenAI API, just changing the Spring AI configuration is not enough to successfully call Groq.
Context
I'd like to be able to call Groq's API using Spring AI. I tried with the following configuration:
I get the following exception when I attempt to make a chat completion call: