Generation Error with Ollama - Inconsistent Output Generation

[juanjuanignacio]

Hi,

I'm experiencing issues while running GEMMA2 on Ollama. Specifically, I'm encountering the following problems:

Error on Message Generation: Whenever a new chat is created, every message results in the error:

Error: Generation failed, in the back end

No output is generated,on the front end.

Inconsistent Message Handling: After retrying the same message multiple times (ranging from 2 to 15 attempts), the message is eventually processed correctly and the output is displayed on the front end.

Server Responsiveness: Despite the above issues, the server responds to every query.

Expected Behavior: Messages should be processed and output generated on the first attempt without errors.

Additional Context:

Ollama Version: 0.3.3
GEMMA2:2b (I've tried others models and the problem is the same)
Operating System: CentOS

Relevant Logs: error message:

    ERROR (537688): Generation failed
        err: {
          "type": "Error",
          "message": "Generation failed",
          "stack":
              Error: Generation failed
                  at Module.generateFromDefaultEndpoint (/chat-ui/src/lib/server/generateFromDefaultEndpoint.ts:23:9)
                  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                  at async generateTitle (/chat-ui/src/lib/server/textGeneration/title.ts:54:10)
                  at async Module.generateTitleForConversation (/chat-ui/src/lib/server/textGeneration/title.ts:17:19)

Its something with the title of the conversation but retrying the message finally the conversations name is changed too. And messages after conversations name is changed have the same problem, rarely it works at first attempt.

My env.local:

  MONGODB_URL="mongodb://localhost:27017"
  HF_TOKEN=Mytoken
  OPENAI_API_KEY="ollama"
  MODELS=`[
    {
        "name": "google/gemma-2-2b-it",
        "chatPromptTemplate": "{{#each messages}}{{#ifUser}}<start_of_turn>user\n{{#if @first}}{{#if @root.preprompt}}{{@root.preprompt}}\n{{/if}}{{/if}}{{content}}<end_of_turn>\n<start_of_turn>model\n{{/ifUser}}{{#ifAssistant}}{{content}}<end_of_turn>\n{{/ifAssistant}}{{/each}}",
        "parameters": {
          "temperature": 0.1,
          "top_p": 0.95,
          "repetition_penalty": 1.2,
          "max_new_tokens": 2048,
          "stop": ["<end_of_turn>"]
        },
        "endpoints": [
          {
           "type": "ollama",
           "baseURL": "http://127.0.0.1:11434",
            "ollamaName" : "gemma2:2b"
          }
        ]
    },
  ]`

  USE_LOCAL_WEBSEARCH=true

Any assistance in resolving this issue would be greatly appreciated. Thank you!

[juanjuanignacio]

Solution Found: I managed to resolve the inconsistency in message generation by switching the model to use the OpenAI API as follows(Ollama supports Open AI API calls):

    {
      "name": "gemma2:2b",
      "chatPromptTemplate": "{{#each messages}}{{#ifUser}}<start_of_turn>user\n{{#if @first}}{{#if @root.preprompt}}{{@root.preprompt}}\n{{/if}}{{/if}}{{content}}<end_of_turn>\n<start_of_turn>model\n{{/ifUser}}{{#ifAssistant}}{{content}}<end_of_turn>\n{{/ifAssistant}}{{/each}}",
      "parameters": {
        "temperature": 0.1,
        "top_p": 0.95,
        "repetition_penalty": 1.0,
        "max_new_tokens": 2048,
        "stop": ["<end_of_turn>", "<start_of_turn>"]
      },
      "endpoints": [
        {
          "type": "openai",
          "baseURL": "http://127.0.0.1:11434/v1",
          "model": "gemma2:2b"
        }
      ]
    }

Persistent Issue: Despite this fix, I am still encountering the error Generation failed in the back end related to the titles of the conversations. The output is consistent and messages are printed correctly on the first attempt, but the conversation title never updates and remains as 'new chat'.

Any assistance in resolving the issue with the conversation titles would be greatly appreciated. Thank you!