search

jankais3r / LLaMA_MPS

Run LLaMA (and Stanford-Alpaca) inference on Apple Silicon GPUs.
GNU General Public License v3.0
583 stars 47 forks source link

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 #1

Closed fdstevex closed 1 year ago

fdstevex commented 1 year ago

Hi - Thanks for building this, it looks like a great way to try out the model.

I wasn't able to follow the instructions exactly - pip3 install -r requirements.txt reported No matching distribution found for torch. The python I have installed is 3.11, so I'm explicitly using pip3.9 / python3.9. I don't know if this is related.

Anyway, when I run the example chat command, I get prompted for my input, and when I enter it, about 30 seconds later, I get this:

Traceback (most recent call last):
  File "/Users/stevex/temp/llama/LLaMA_MPS/chat.py", line 130, in <module>
    fire.Fire(main)
  File "/opt/homebrew/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/homebrew/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/homebrew/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/stevex/temp/llama/LLaMA_MPS/chat.py", line 113, in main
    results = generator.generate(
  File "/Users/stevex/temp/llama/LLaMA_MPS/llama/generation.py", line 63, in generate
    next_token = torch.multinomial(
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

Using a Mac Studio, 32gb RAM.

jankais3r commented 1 year ago

Hi,

Very strange that the torch installation failed… Could you double check whether you are running arm64 version of Python and not x86 under Rosetta?

What's the output of the following command?

file `which python3`

Alternatively, could you also try with Python 3.11?

If that doesn't help, to help you figure out what's going on, please repeat all steps from the readme, and reply here with the complete output from your terminal.

fdstevex commented 1 year ago

Hi, thanks for the response .. I've been learning how to manage the python environment and the changes I made to get llama.cpp to work have also fixed my problem here. I'm not sure which change, but when I tried to get it to work today, it worked fine (although very slowly). So it's not LLaMA_MPS, it was my setup. Thanks!