search

kuvaus / LlamaGPTJ-chat

Simple chat program for LLaMa, GPT-J, and MPT models.
MIT License
217 stars 50 forks source link

bug: Seed parameter doesn't work. #14

Closed saul-jb closed 1 year ago

saul-jb commented 1 year ago

It appears the seed parameter doesn't work:

./chat -m ggml-gpt4all-j-v1.3-groovy.bin -s 1

I can provide it with the same prompt (What is a dog?) on two different clean sessions and the response may be different. I would have expected the exact same result for model + template + context + seed + prompt combination.

Have I missed something else that might cause different results or is seed not working?

saul-jb commented 1 year ago

As far as I can see in the code the seed parameter only affects the --random-prompt parameter. If that is the case then it should probably be mentioned as part of the help.

Is there any way to produce deterministic results?

kuvaus commented 1 year ago

Temperature, not seed controls the prompt token generation:

So just set temperature to zero. But many models produce odd results with --temp 0, so set it to a very low number instead: --temp 0.001

I added a comment in -h about the seed. :) (I think it can be used on some llama models, but current backend does not support that.)

saul-jb commented 1 year ago

So just set temperature to zero. But many models produce odd results with --temp 0, so set it to a very low number instead: --temp 0.001

Thanks, I needed to read up more on LLM parameters, it seems that --top_k 1 can also be used to restrict the model to only choosing the most likely result.