Open intc-hharshtk opened 7 months ago
It looks like the parallel flag regressed in a recent upstream upgrade. I'll leave this open to track progress. It'll likely be fixed on the next sync if not sooner.
yes, same issues , --np doesn't work , and I wonder is there any benchmark about the batch inference.
Is it fixed ?? Which old build have parallel processing working which we can use till we have this issue fixed.
Found a bug related to -np. When -np is set too large or the output content is more longer, it will be triggered, and meaningless content will be output repeatedly until the max_tokens upper limit is reached.
./llamafile --server -m model.gguf -cb -np 20 -c 4096 -ngl 999
I have tried multiple models and the same problem occurs.
When I open two browser window, and enter the prompt, then the output from first window completed and then only the output in other window is generated. Is there any switch that I am missing.
Command : ./llava-v1.5-7b-q4.llamafile -ngl 9999 --server --port 8080 --parallel 5 -t 10