Please handle the case your logits contain nans

turboderp / exllama

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

MIT License

2.74k stars 215 forks source link

Please handle the case your logits contain nans #222

Open ParisNeo opened 1 year ago

ParisNeo commented 1 year ago

Hi, I was prompting llama-2-7B and got into this error.

Can you please handle the case there are nans in the logits?

turboderp commented 1 year ago

NaN or infinities in the logits implies the model has failed for one reason or another. This may be caused by a bug in the implementation, or it may be a corrupted/incorrectly finetuned model, but I'd need a lot more details to start figuring out what the cause is.