search

simonw / llm-gpt4all

Plugin for LLM adding support for the GPT4All collection of models
Apache License 2.0
218 stars 20 forks source link

Empty Response From Gpt4all Meta-Llama-3 (Llama 3.1 8B Instruct 128k) #39

Open stungeye opened 3 months ago

stungeye commented 3 months ago

Issue Summary: Llama 3.1 via gpt4all returns blank responses. Llama 3.0 works fine.

Possible Fix: According to this gpt4all issue, the Llama 3.1 model requires gpt4all version 3.1.1, whereas llm-gpt4all requires gpt4all>=2.5.1. This is a problem for folks installing gpt4all via pip, as the latest pypi version of gpt4all is 2.7.0.

More Details:

When running llm (v0.15) and llm-gpt4all (v0.4) I can download the gpt4all llama 3 and llama 3.1 models.

The version of gpt4all installed on my machine (by way of llm-gpt4all) was 2.6.0, but I can manually upgrade to 2.7.0.

Output from llm models:

<snip>
gpt4all: Meta-Llama-3 - Llama 3.1 8B Instruct 128k, 4.34GB download, needs 8GB RAM (installed)
gpt4all: Meta-Llama-3-8B-Instruct - Llama 3 8B Instruct, 4.34GB download, needs 8GB RAM (installed)
</snip>

This works:

 llm -m Meta-Llama-3-8B-Instruct 'What is the capital of Canada?'
 Ottawa, which is located in the province of Ontario.

This returns a blank response:

llm -m Meta-Llama-3 'What is the capital of Canada?'

Log shows:

# 2024-08-02T13:09:39    conversation: 01j49kbn2wzxgjtbh8v9x2pjth

Model: **Meta-Llama-3**

## Prompt:

What is the capital of Canada?

## Response: