Open Atry opened 2 weeks ago
Note that this bug is specific to meta-llama/Meta-Llama-3-8B-Instruct
. If I replace meta-llama/Meta-Llama-3-8B-Instruct
with kevin009/babyllama-v0.6
, no error would be raised.
Also if I comment out the weight_quantization
configuration, no error would be raised.
Describe the bug I got the error
RuntimeError: probability tensor contains either
inf,
nanor element < 0
when trying to run deepspeed_engine.generate whenMeta-Llama-3-8B-Instruct
is initialized with either 4-bit or 8-bit quantization.To Reproduce
Run the following code
Then the output is
Expected behavior No error
ds_report output
Screenshots Not applicable
System info (please complete the following information):
Launcher context Just
python
cli, notdeepspeed
cli.Docker context Not using Docker
Additional context