matatonic
commented
1 year ago A few things are causing problems here.
1.
Exception: When loading characters/instruction-following/None.yaml: FileNotFoundError(2, 'No such file or directory') Warning: Loaded default instruction-following template for model.
This means the model isn't loading the instruction set for the model. This is a major problem. There isn't a standard file for Beluga yet so you need to create your own. See: https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ#prompt-template-orca-hashes
You can try this PR for a fix: https://github.com/oobabooga/text-generation-webui/pull/3415
-
Warning: $This model maximum context length is 2048 tokens.
StableBeluga2 is 4k, see the above PR for a fix to this also.
3.
However, your messages resulted in over 91 tokens and max_tokens is 32768.
max_tokens is reserved from the output, try removing max_tokens if you want to use all available context, or set it to a smaller value like 1000.
RazeLighter777
Describe the bug
Always got 0 tokens output generated when run HuggingFace.co/TheBloke_StableBeluga2-70B-GPTQ
Load model and try to run it, with a simple request, it always return 0 tokens and generated nothing.
Environment:
python server.py --verbose --model-dir /mnt/c/TechBuilder/HuggingFace.co --model TheBloke_StableBeluga2-70B-GPTQ --extensions openai --loader autogptq --auto-devices --gpu-memory 23 --load-in-4bit --triton --no_use_cuda_fp16 --listen --listen-host 0.0.0.0 --listen-port 7860
No settings.yaml configured.
after server start, try to raise a request via Postman:
{ "model": "", "max_tokens": 32768, "temperature": 0, "top_p": 1, "messages": [ { "content": "You are a helpful assistant to support my writing.", "role": "system" }, { "content": "Develop a outline for a children story. In the story, a boy adventured into a fairy kindom and experieced many things. Give about 250 words.", "role": "user" } ] }
Then returned:
{ "id": "chatcmpl-1690983042630880000", "object": "chat.completions", "created": 1690983042, "model": "TheBloke_StableBeluga2-70B-GPTQ", "choices": [ { "index": 0, "finish_reason": "stop", "message": { "role": "assistant", "content": "" } } ], "usage": { "prompt_tokens": 91, "completion_tokens": 1, "total_tokens": 92 } }
The content returned is blank. And at server side, got message:
Warning: $This model maximum context length is 2048 tokens. However, your messages resulted in over 91 tokens and max_tokens is 32768.
You are a helpful assistant to support my writing. You are a helpful assistant. Answer as concisely as possible. User: I want your assistance. Assistant: Sure! What can I do for you? User: Develop a outline for a children story. In the story, a boy adventured into a fairy kindom and experieced many things. Give about 250 words. Assistant:
Output generated in 1.82 seconds (0.00 tokens/s, 0 tokens, context 91, seed 1657817950) 172.20.112.1 - - [02/Aug/2023 21:30:44] "POST /v1/chat/completions HTTP/1.1" 200 -
Even I removed the max_tokens in the request body, still output nothing.
Btw, using the same configuration, to run the model: TheBloke_StableBeluga-13B-GPTQ, it could work properly.
Is there an existing issue for this?
Reproduction
1, load model using cmd: python server.py --verbose --model-dir /mnt/c/TechBuilder/HuggingFace.co --model TheBloke_StableBeluga2-70B-GPTQ --extensions openai --loader autogptq --auto-devices --gpu-memory 23 --load-in-4bit --triton --no_use_cuda_fp16 --listen --listen-host 0.0.0.0 --listen-port 7860
2, send request via postman using openai api schema:
3, got nothing in the content field.
Screenshot
No response
Logs
System Info