Issue: Llama 2 loaded, but seems dumber?

[enzyme69]

Issue you'd like to raise.

I am using Llama 2 model that I got from here: https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2167b5d129

When running using above command, the Llama model seems to be smarter.

Compared to the one I am running via gpt4all interface. I wonder why? Is there a setting or temperature I need to adjust manually?

echo "Prompt: " \
    && read PROMPT \
    && ./main \
        -t 8 \
        -ngl 1 \
        -m llama-2-13b-chat.ggmlv3.q4_0.bin \
        --color \
        -c 2048 \
        --temp 0.7 \
        --repeat_penalty 1.1 \
        -n -1 \
        -p "[INST] <<SYS>> ${SYSTEM} <</SYS>> ${PROMPT} [/INST]"

Suggestion:

No response

[qnixsynapse]

I use this prompt template:

And using your prompt it responds like this:

[niansa]

Additionally, gpt4all does not currently set the rhl_eps param of llama.cpp to the correct value for llama2.

[qnixsynapse]

@niansa rhl_eps? You mean the eps value in RMSNorm layer?