macbook M2 runtime MPS issue

[sadesguy]

I am on the latest MacOs (13.4 (22f66)) on a MacBook air m2 16gb. I installed the opt6.7B version whilst choosing to run on CPU (also tried apple silicon, got same results). The issue occurs when I am in the localhost and when I press "Generate" with any prompt, I also enabled --mlock option, no other differences from default configuration.

What can cause this issue, and are there any fixes for it?

Traceback of the issue:

/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py:719: UserWarning: MPS: no support for int64 repeats mask, casting it to int32 (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1682343686130/work/aten/src/ATen/native/mps/operations/Repeat.mm:236.)
  input_ids = input_ids.repeat_interleave(expand_size, dim=0)
Traceback (most recent call last):
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/text-generation-webui/modules/callbacks.py", line 73, in gentask
    ret = self.mfunc(callback=_callback, **self.kwargs)
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/text-generation-webui/modules/text_generation.py", line 277, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 1568, in generate
    return self.sample(
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2615, in sample
    outputs = self(
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 945, in forward
    outputs = self.model.decoder(
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 655, in forward
    pos_embeds = self.embed_positions(attention_mask, past_key_values_length)
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/sadesguy/Downloads/one-click-installers-main/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 115, in forward
    positions = (torch.cumsum(attention_mask, dim=1).type_as(attention_mask) * attention_mask).long() - 1
RuntimeError: MPS does not support cumsum op with int64 input
Output generated in 4.72 seconds (0.00 tokens/s, 0 tokens, context 63, seed 1888692510)

[simonpeters]

I think you need to update OS X

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.