search

SafeAILab / EAGLE

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)
https://arxiv.org/pdf/2406.16858
Apache License 2.0
780 stars 79 forks source link

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 #45

Closed cyLi-Tiger closed 7 months ago

cyLi-Tiger commented 7 months ago

Great work!

I tried your example for llama-7b-chat and changed the tree structure in choices.py into baseline = [[0], [0, 0], [0, 0, 0], [0, 0, 0, 0]] to simulate speculative decoding without token tree, and got the error above. Any clues?

log attached: image

Liyuhui-12 commented 7 months ago

This seems to be an issue with the original LLaMA2-Chat encountering NaN. What was your input?

cyLi-Tiger commented 7 months ago

This seems to be an issue with the original LLaMA2-Chat encountering NaN. What was your input?

Simply "Hello". @Liyuhui-12

Liyuhui-12 commented 7 months ago

I can run this code normally. Did you use the correct weights for LLaMA2-Chat? It should be meta-llama/Llama-2-7b-chat-hf instead of meta-llama/Llama-2-7b-chat.

cyLi-Tiger commented 7 months ago

I got the tree shape wrong, my bad. Thanks!

reyna-abhyankar commented 6 months ago

What was wrong with your original tree shape @cyLi-Tiger? I am facing a similar issue