search

Noble-Lab / casanovo

De Novo Mass Spectrometry Peptide Sequencing with a Transformer Model
https://casanovo.readthedocs.io
Apache License 2.0
116 stars 40 forks source link

Rank of updates array (1) must be greater than or equal to inner-most dimension of indices array (29) #285

Open wsnoble opened 9 months ago

wsnoble commented 9 months ago

On an M2 Mac, I am getting the error shown below. I do have the fallback variable set properly:

% echo $PYTORCH_ENABLE_MPS_FALLBACK
1

I am using the following version:

Casanovo: 4.0.1
Depthcharge: 0.2.3
Lightning: 2.1.3
PyTorch: 2.1.2

The input MGF file is 151009_exo4_4.short.mgf.txt

% casanovo sequence 151009_exo4_4.short.mgf 
WARNING: Dataloader multiprocessing is currently not supported on Windows or MacOS; using only a single thread.
Seed set to 454
INFO: Model weights file /Users/wnoble/Library/Caches/casanovo/casanovo_massivekb_v4_0_0.ckpt retrieved from local cache
INFO: Casanovo version 4.0.1
INFO: Sequencing peptides from:
INFO:   151009_exo4_4.short.mgf
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
INFO: Reading 1 files...
151009_exo4_4.short.mgf: 2000spectra [00:00, 20073.48spectra/s]
WARNING: Skipped 1282 spectra with invalid precursor info
Predicting DataLoader 0:   0%|                            | 0/1 [00:00<?, ?it/s]WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSNDArray/Kernels/MPSNDArrayGatherND.mm:256: failed assertion `Rank of updates array (1) must be greater than or equal to inner-most dimension of indices array (29)'
zsh: abort      casanovo sequence 151009_exo4_4.short.mgf
/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
wsnoble commented 9 months ago

Update: this error does not appear when I don't use the GPU.

loganjohnson0 commented 1 week ago

Excited to try out Casanovo and thanks for the great documentation. I also have a similar error, which seems to impact M series Macs in other pipelines as well, likely related to lack of memory or memory allocation. My process was killed almost immediately after the start of the search on the example mgf file from the documetation. Looking forward to your future developments!

Apple M2 Pro
16 GB Memory 
MacOS Sequoia 15.1

(casanovo_env) MacBook-Pro-2:~ logan.johnson$ conda list python
# packages in environment at /Users/logan.johnson/opt/anaconda3/envs/casanovo_env:
#
# Name                    Version                   Build  Channel
python                    3.10.15              hce00570_1  

(casanovo_env) MacBook-Pro-2:~ logan.johnson$ echo $PYTORCH_ENABLE_MPS_FALLBACK
1
(casanovo_env) MacBook-Pro-2:~ logan.johnson$ casanovo sequence /Users/logan.johnson/Downloads/sample_preprocessed_spectra.mgf
WARNING: Dataloader multiprocessing is currently not supported on Windows or MacOS; using only a single thread.
Seed set to 454
INFO: Model weights file /Users/logan.johnson/Library/Caches/casanovo/casanovo_v4_2_0_v4_2_0.ckpt retrieved from local cache
INFO: Casanovo version 4.2.1
INFO: Sequencing peptides from:
INFO:   /Users/logan.johnson/Downloads/sample_preprocessed_spectra.mgf
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
INFO: Reading 1 files...
/Users/logan.johnson/Downloads/sample_preprocessed_spectra.mgf: 128spectra [00:00, 13661.88spectra/s]
Predicting DataLoader 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
/AppleInternal/Library/BuildRoots/4b66fb3c-7dd0-11ef-b4fb-4a83e32a47e1/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSNDArray/Kernels/MPSNDArrayGatherND.mm:255: failed assertion `Rank of updates array (1) must be greater than or equal to inner-most dimension of indices array (29)'
Abort trap: 6
(casanovo_env) MacBook-Pro-2:~ logan.johnson$ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

casanovo_20241112120802.log

wsnoble commented 1 week ago

FWIW, I upgraded to v4.2.1 and ran the test above on my M2, but I get a different error. I'm on Python 3.10.12.

>  casanovo sequence sample_preprocessed_spectra.mgf
WARNING: Dataloader multiprocessing is currently not supported on Windows or MacOS; using only a single thread.
Seed set to 454
INFO: Downloading model weights file /Users/wnoble/Library/Caches/casanovo/casanovo_v4_2_0_v4_2_0.ckpt from https://github.com/Noble-Lab/casanovo/releases/download/v4.2.0/casanovo_v4_2_0.ckpt
100%|████████████████████████████████████████| 542M/542M [02:20<00:00, 4.06MB/s]
INFO: Casanovo version 4.2.1
INFO: Sequencing peptides from:
INFO:   sample_preprocessed_spectra.mgf
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
WARNING: /opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/lightning/pytorch/utilities/migration/utils.py:55: The loaded checkpoint was produced with Lightning v2.2.1, which is newer than your current Lightning version: v2.1.3

WARNING: /opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/lightning/pytorch/utilities/migration/utils.py:55: The loaded checkpoint was produced with Lightning v2.2.1, which is newer than your current Lightning version: v2.1.3

INFO: Reading 1 files...
sample_preprocessed_spectra.mgf: 0spectra [00:00, ?spectra/s]
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/bin/casanovo", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/rich_click/rich_command.py", line 126, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/casanovo/casanovo.py", line 143, in sequence
    runner.predict(peak_path, output)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/casanovo/denovo/model_runner.py", line 160, in predict
    test_index = self._get_index(peak_path, False, "")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/casanovo/denovo/model_runner.py", line 394, in _get_index
    return Index(index_fname, filenames, valid_charge=valid_charge)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/depthcharge/data/hdf5.py", line 104, in __init__
    self.add_file(ms_file)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/depthcharge/data/hdf5.py", line 192, in add_file
    parser.read()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/depthcharge/data/parsers.py", line 98, in read
    self.mz_arrays = np.concatenate(self.mz_arrays).astype(np.float64)
  File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate