Closed fenixlam closed 3 weeks ago
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
This issue was closed because it has been stalled for 5 days with no activity.
I copy the example python and run it. The only thing I change is comment the openai API and release local server setting:
it is a koboldcpp server, run with meta llama 3 70B Q4XS gguf model.
After I run the program, it get the answer:
yes, it stop after it generates 100 tokens every time. Clearly, the program didn't submit max_token parameter to koboldcpp server, so it didn't generate more than 100 tokens. Koboldcpp didn't have initialize parameter to config the response token in its initial setting, it is submitted in request, and I didn't find where I can change the code to add the max_token parameter in the request.