Open ZhangYuanhan-AI opened 5 months ago
Also been getting this, both using transformers directly as you've tried, and also through FastChat.
This appears to be a bug with transformers, as I get it on 4.38.1 and 4.38.2, but it works fine on v4.31.0 (the first version to add RoPE scaling). Weridly though I can't find anything in the transformers changelog referring to RoPE until 4.38.2, but it's also broken in 4.38.1.
How can I solve it
You can solve this by downgrading the transformers library to 4.31.0 (or possibly later, but 4.38.1 and 4.38.2 are definitely broken). Setting max_new_token=128
won't solve it, it will just take longer to break as it's generating fewer tokens at once.
This is my demo code:
This is the error code:
It seems that the maximum size of the causal mask is 8196: https://github.com/huggingface/transformers/blob/0290ec19c901adc0f1230ebdccad11c40af026f5/src/transformers/models/llama/modeling_llama.py#L1079
env: transformer: 4.38.2