Open d-01 opened 8 months ago
Thanks! Will look into this. Do you happen to have llama.cpp installed as well? If so it would be nice to know if the same issue happens there or seems to be specific to HuggingFace.
Update: I tried this with Mistral 7B with no issues. Long story, but I have constraints that prevent me from running the llama based vicuna version right now, if you find a similar issue with a non-llama model that would also be helpful.
Thanks for your comment on this issue.
Code for llama.cpp experiment:
import guidance
from guidance import gen, select, substring, models
# https://huggingface.co/TheBloke/vicuna-13B-v1.5-GGUF/blob/main/vicuna-13b-v1.5.Q8_0.gguf
MODEL_PATH = '/home/user/work/hf-models/vicuna-13b-v1.5.Q8_0.gguf'
lm = models.LlamaCpp(MODEL_PATH, n_gpu_layers=-1, n_ctx=2048)
def complete(prompt, options=[]):
if options:
options = [' ' + x + '\n' for x in options]
generation = lm + prompt + select(options, name='output')
else:
generation = lm + prompt + gen(name='output', stop='\n', max_tokens=400, temperature=0)
output = generation.get('output', '[ERROR] "output" variable is missing').strip()
return output
...
The bug
select
generates incorrect output, while unconstrained generation (gen
) is correct.Description It seems the problem somehow connected to 8-bit quantization, because I was unable to reproduce the problem in other modes (fp16 non-quantized and 4-bit quantized).
Given this kind of prompt:
_j
.Free (unconstrained) generation gives correct answer:
{'▁_': 903}
is the token with the maximum probability.But guided (constrained) generation in this case gives wrong answer, which in theory should be impossible, given most probable token must be the same in both cases:
{'▁w': 281}
was chosen over the token{'▁_': 903}
with the maximum probability?Note that token healing should not be involved, since the prompt ends with a valid (non-partial) token
{'▁=': 353}
.Probably related issue: https://github.com/guidance-ai/guidance/issues/472
To Reproduce
System info (please complete the following information): Environment:
accelerate==0.22.0
bitsandbytes==0.41.1
guidance==0.1.5
numpy==1.24.4
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
sentencepiece==0.1.99
tokenizers==0.13.3
torch==1.13.0
transformers==4.33.1