search

Mozilla-Ocho / llamafile

Distribute and run LLMs with a single file.
https://llamafile.ai
Other
17.85k stars 898 forks source link

Commandline options in server mode #274

Open dezifit opened 4 months ago

dezifit commented 4 months ago

I'm trying to reproduce examples of EM German with llamafile v0.6.2 in server mode.

The example page lists the used options and results for different models. While llamafile's usage help has the most options in it's "common" section, it doesn't accept the following invocation:

./llamafile -m em_german_mistral_v01.Q4_K_M.gguf -spf system_prompt_de.json --server --n-predict 2000 --temp 0.7 --frequency-penalty 0.1 --top-k 15

and fails with various "unknown argument:" errors.

Invocation with model and system-prompt specification only is accepted but seems to ignore the system prompt (prompt, user and bot name in the start/setup page aren't changed, option --prompt is accepted in cli mode only).

Is it possible to accept some more options (for all parameters available in the startup page) in server mode?

woheller69 commented 3 months ago

llamafile seems to ignore some commandline parameters, e.g. -n or --n-predict and others