search

Mozilla-Ocho / llamafile

Distribute and run LLMs with a single file.
https://llamafile.ai
Other
20.63k stars 1.04k forks source link

Bug: Unable to specify seed from cli #566

Open mjpowersjr opened 2 months ago

mjpowersjr commented 2 months ago

Contact Details

mjpowersjr@gmail.com

What happened?

When attempting to use the -seed argument, e.g.

./mistral-7b-instruct-v0.2.Q4_0.llamafile --temp 0.7 -seed -1 -p '[INST]Write a story about llamas[/INST]'

I receive the following error:

error: unknown argument: -seed

I've tried this with a few different models, same results. If this is expected behavior, maybe updating the help documentation with more details about when the --seed argument can be used would be helpful?

Thanks!

Version

llamafile v0.8.5

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

belisarius222 commented 2 months ago

@jalehman and I just reproduced this on MacOS, but changing the argument to --seed instead of -seed fixed it, so I don't think this is unintended behavior.

mjpowersjr commented 2 months ago

@belisarius222 - Thanks for looking into this - just realized I missed the second dash in the submitted example. I actually tried a few other combinations of arguments including --seed before ending up here.

  1. It seems like --seed is only intended to be used when supplying a prompt via the command line (not when using the --server flag. It might be nice to include a hint about that in the docs or error msg.)

  2. After a second look, I think there are still some edge cases with argument parsing. It seems using the full form --prompt argument trips up the parser.

Works!

./mistral-7b-instruct-v0.2.Q4_0.llamafile --temp 0.7 --seed -1 -p '[INST]Write a story about llamas[/INST]'

Fails

./mistral-7b-instruct-v0.2.Q4_0.llamafile --temp 0.7 --seed -1 --prompt '[INST]Write a story about llamas[/INST]'

error: unknown argument: --temp

Fails

./mistral-7b-instruct-v0.2.Q4_0.llamafile --prompt '[INST]Write a story about llamas[/INST]'

error: unknown argument: --prompt
Djip007 commented 2 months ago

Mistral-7B-Instruct-v0.2-llamafile This model is "old" use an old version of llamafile... did you have the same with https://huggingface.co/Mozilla/Mistral-7B-Instruct-v0.3-llamafile/tree/main

you can try with the last llamafile from: https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.8.13 too

(ref: https://github.com/Mozilla-Ocho/llamafile/discussions/543#discussioncomment-10380479)