Groq fails with: Failed to decode JSON output

[glindberg2000]

Describe the bug When using the Groq endpoint with MemGPT, I encounter an error that prevents successful conversation completion. The error arises during the parsing of JSON output from the local language model (LLM). The error message indicates a failure to decode valid MemGPT JSON from the LLM output.

Please describe your setup

• [x] How did you install memgpt?
  • Pip install (pip install pymemgpt) and Docker.
• [x] Describe your setup
  • OS: Linux
  • Running memgpt in both pip-installed and Docker environments.
• [x] MemGPT Config

DB: Postgres from local and memgpt Docker image, tried sqlite3 embedding_endpoint = https://embeddings.memgpt.ai embedding_model = BAAI/bge-large-en-v1.5 Model: llama3 7B and 70B

Screenshots

Additional context The error occurs consistently after a few (sometimes longer) rounds of conversation using the Groq endpoint. The conversation starts successfully but fails during the parsing of JSON output from the LLM especially once it starts using function calls or getting near the context limit but the token limit errors (429) are separate from this JSON parsing error.

---

Local LLM details

• [x] The exact model you're trying to use: Groq endpoint
• [x] The local LLM backend you are using: Local configuration
• [x] Your hardware for the local LLM backend: Local computer with Linux operating system

[sarahwooders]

I believe this is a model issue - we are tying to work with better errors for model failures.

[cpacker]

Could potentially be fixed by #1257