YASS-base container doesn't work

[rat-h]

My colleague and I are trying to find a good spike sorter for our very unusual data. We have tried a bench of them: kilosort 2.0, kilosort 2.5, kilosort 30, tridesclous, spykingcircus, ironclust - all in containerized version.

recording_saved = si.load_extractor("preprocessed")
ks20_sorting = si.run_kilosort2(recording=recording_saved, output_folder="ks20",singularity_image="spikeinterface/kilosort2-compiled-base:latest")
ks20_sorting.save(folder="ks20-sorting")
ks20_we = si.extract_waveforms(
    recording_saved, ks20_sorting, 'ks20-waveforms',
    max_spikes_per_unit=500,
    ms_before=1.5, ms_after=2.5,
    n_jobs=16, total_memory='10G',
    progress_bar=True,
)

We were able to run all of these spike sorters without any problems and compare results!

However, yass container doesn't work and returns an error message.

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:3d07ae8a5cfae06e8b59657be91de80f8b1756cac34f33093164d5b6f27b7a60
Copying blob sha256:0aaee4d952f0088f1c397964693a99ed7a8e809a7f087a2456c101e7d3bd5df5
Copying blob sha256:08f9fb368cf50d6fee11f05a3683312e842188c4660f72b98e8b728ae483b63e
Copying blob sha256:11323ed2c65349758e68a03a8e43825ec263dc9790daea93cf83b18ad0703109
Copying blob sha256:003bcea46c04fbe5893ee7c6bc276fe337f2ac00f34c75501c5f4871ed26c7a1
Copying blob sha256:b4c05861b4dfa58a080ee2454b61d3caf1d1320eea61e0a655d51739019cae2a
Copying blob sha256:93e0937bce3de61746cd07828ac14bbfd6be2543b61c2cb36591693a806f6310
Copying blob sha256:a818ff5e3d218ac3e88f5f7d5492e35cdb9d041978d004463f9d2639670be6b8
Copying blob sha256:7e4a2bed2378dd30858a12c1b12d05ef404139edc37e3a6a4cbb491e6ccf5039
Copying blob sha256:af3ca23b3e56f41b70530e7f13e333b5c412e5c361bc55f587ddcde5d2bf8ac5
Copying blob sha256:8fe3bf88b8a2ead72c89d699a1f055f55a8aa108b611e5366834546fc589baa4
Copying blob sha256:7d94343897f22a68f039f283f56c631d3d8840aea350b3ae18bff1174f85b087
Copying blob sha256:09949d2afee007e3aa3892a4afd94ed8a9b7e848ebb6fddd62a08a52b52faec2
Copying blob sha256:e3f6e4676372867d91234c8fd1ac0a7dd3c45f4ef7230943ec4419f7108f98ac
Copying blob sha256:c1e66fd9086256245bc314a9c49c3ccea2584069cbaf5202d6705c61f4107a7d
Copying blob sha256:00f9fc2ec83e80c4973839ce6a6b3662fecbc32f331266ed26a700ad2c828694
Copying blob sha256:d1ff7f4ba7b321e913ef7f23f42f151b0dedd7c1e256df179d11d832e0f55338
Copying blob sha256:04a6139bd10d52ef01dd3feb018ddc46522878c47e80c7fad60a4d80a9890dd9
Copying blob sha256:bcfbc387c0656eba0cdb459dfd30daec7570465ebecce4d5ddebb0f0972c7c7c
Copying blob sha256:4cdecdba6820e97126f6e9e512d20da229cdaee5d5942ac0643dbdd7d1fc561d
Copying blob sha256:71e92c1ce896438c6d4de350fce4d7ac7e6234f96ee297d056daa42156f5c671
Copying blob sha256:35458b947e57b18e0c39801519885f99ca6a55ec01eb83452f438e531460107b
Copying blob sha256:5a9cb6c9cef30b98f8ad331e22dad67eb7d9297967b699221527ff1b6e8c4f84
Copying blob sha256:322f61cc85dfc15de2bdca3dfd60b88e01e883527ac8fa3095584e45051ff433
Copying config sha256:8c1170d5cb783868b125c0d5e66c654580db7621b46e9dd0cfa2f727d5ec4ab7
Writing manifest to image destination
Storing signatures
2023/01/06 15:51:32  info unpack layer: sha256:11323ed2c65349758e68a03a8e43825ec263dc9790daea93cf83b18ad0703109
2023/01/06 15:51:33  info unpack layer: sha256:0aaee4d952f0088f1c397964693a99ed7a8e809a7f087a2456c101e7d3bd5df5
2023/01/06 15:51:33  info unpack layer: sha256:003bcea46c04fbe5893ee7c6bc276fe337f2ac00f34c75501c5f4871ed26c7a1
2023/01/06 15:51:33  info unpack layer: sha256:08f9fb368cf50d6fee11f05a3683312e842188c4660f72b98e8b728ae483b63e
2023/01/06 15:51:33  info unpack layer: sha256:b4c05861b4dfa58a080ee2454b61d3caf1d1320eea61e0a655d51739019cae2a
2023/01/06 15:51:33  info unpack layer: sha256:3d07ae8a5cfae06e8b59657be91de80f8b1756cac34f33093164d5b6f27b7a60
2023/01/06 15:51:40  info unpack layer: sha256:93e0937bce3de61746cd07828ac14bbfd6be2543b61c2cb36591693a806f6310
2023/01/06 15:51:40  info unpack layer: sha256:a818ff5e3d218ac3e88f5f7d5492e35cdb9d041978d004463f9d2639670be6b8
2023/01/06 15:51:52  info unpack layer: sha256:7e4a2bed2378dd30858a12c1b12d05ef404139edc37e3a6a4cbb491e6ccf5039
2023/01/06 15:51:52  info unpack layer: sha256:af3ca23b3e56f41b70530e7f13e333b5c412e5c361bc55f587ddcde5d2bf8ac5
2023/01/06 15:52:00  info unpack layer: sha256:8fe3bf88b8a2ead72c89d699a1f055f55a8aa108b611e5366834546fc589baa4
2023/01/06 15:52:00  info unpack layer: sha256:7d94343897f22a68f039f283f56c631d3d8840aea350b3ae18bff1174f85b087
2023/01/06 15:52:00  info unpack layer: sha256:09949d2afee007e3aa3892a4afd94ed8a9b7e848ebb6fddd62a08a52b52faec2
2023/01/06 15:52:00  info unpack layer: sha256:e3f6e4676372867d91234c8fd1ac0a7dd3c45f4ef7230943ec4419f7108f98ac
2023/01/06 15:52:00  info unpack layer: sha256:c1e66fd9086256245bc314a9c49c3ccea2584069cbaf5202d6705c61f4107a7d
2023/01/06 15:52:00  info unpack layer: sha256:00f9fc2ec83e80c4973839ce6a6b3662fecbc32f331266ed26a700ad2c828694
2023/01/06 15:52:00  info unpack layer: sha256:d1ff7f4ba7b321e913ef7f23f42f151b0dedd7c1e256df179d11d832e0f55338
2023/01/06 15:52:00  info unpack layer: sha256:04a6139bd10d52ef01dd3feb018ddc46522878c47e80c7fad60a4d80a9890dd9
2023/01/06 15:52:02  info unpack layer: sha256:bcfbc387c0656eba0cdb459dfd30daec7570465ebecce4d5ddebb0f0972c7c7c
2023/01/06 15:52:06  info unpack layer: sha256:4cdecdba6820e97126f6e9e512d20da229cdaee5d5942ac0643dbdd7d1fc561d
2023/01/06 15:52:06  info unpack layer: sha256:71e92c1ce896438c6d4de350fce4d7ac7e6234f96ee297d056daa42156f5c671
2023/01/06 15:52:06  info unpack layer: sha256:35458b947e57b18e0c39801519885f99ca6a55ec01eb83452f438e531460107b
2023/01/06 15:52:34  info unpack layer: sha256:5a9cb6c9cef30b98f8ad331e22dad67eb7d9297967b699221527ff1b6e8c4f84
2023/01/06 15:52:39  info unpack layer: sha256:322f61cc85dfc15de2bdca3dfd60b88e01e883527ac8fa3095584e45051ff433
INFO:    Creating SIF file...
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:3d07ae8a5cfae06e8b59657be91de80f8b1756cac34f33093164d5b6f27b7a60
Copying blob sha256:0aaee4d952f0088f1c397964693a99ed7a8e809a7f087a2456c101e7d3bd5df5
Copying blob sha256:08f9fb368cf50d6fee11f05a3683312e842188c4660f72b98e8b728ae483b63e
Copying blob sha256:11323ed2c65349758e68a03a8e43825ec263dc9790daea93cf83b18ad0703109
Copying blob sha256:003bcea46c04fbe5893ee7c6bc276fe337f2ac00f34c75501c5f4871ed26c7a1
Copying blob sha256:b4c05861b4dfa58a080ee2454b61d3caf1d1320eea61e0a655d51739019cae2a
Copying blob sha256:93e0937bce3de61746cd07828ac14bbfd6be2543b61c2cb36591693a806f6310
Copying blob sha256:a818ff5e3d218ac3e88f5f7d5492e35cdb9d041978d004463f9d2639670be6b8
Copying blob sha256:7e4a2bed2378dd30858a12c1b12d05ef404139edc37e3a6a4cbb491e6ccf5039
Copying blob sha256:af3ca23b3e56f41b70530e7f13e333b5c412e5c361bc55f587ddcde5d2bf8ac5
Copying blob sha256:8fe3bf88b8a2ead72c89d699a1f055f55a8aa108b611e5366834546fc589baa4
Copying blob sha256:7d94343897f22a68f039f283f56c631d3d8840aea350b3ae18bff1174f85b087
Copying blob sha256:09949d2afee007e3aa3892a4afd94ed8a9b7e848ebb6fddd62a08a52b52faec2
Copying blob sha256:e3f6e4676372867d91234c8fd1ac0a7dd3c45f4ef7230943ec4419f7108f98ac
Copying blob sha256:c1e66fd9086256245bc314a9c49c3ccea2584069cbaf5202d6705c61f4107a7d
Copying blob sha256:00f9fc2ec83e80c4973839ce6a6b3662fecbc32f331266ed26a700ad2c828694
Copying blob sha256:d1ff7f4ba7b321e913ef7f23f42f151b0dedd7c1e256df179d11d832e0f55338
Copying blob sha256:04a6139bd10d52ef01dd3feb018ddc46522878c47e80c7fad60a4d80a9890dd9
Copying blob sha256:bcfbc387c0656eba0cdb459dfd30daec7570465ebecce4d5ddebb0f0972c7c7c
Copying blob sha256:4cdecdba6820e97126f6e9e512d20da229cdaee5d5942ac0643dbdd7d1fc561d
Copying blob sha256:71e92c1ce896438c6d4de350fce4d7ac7e6234f96ee297d056daa42156f5c671
Copying blob sha256:35458b947e57b18e0c39801519885f99ca6a55ec01eb83452f438e531460107b
Copying blob sha256:5a9cb6c9cef30b98f8ad331e22dad67eb7d9297967b699221527ff1b6e8c4f84
Copying blob sha256:322f61cc85dfc15de2bdca3dfd60b88e01e883527ac8fa3095584e45051ff433
Copying config sha256:8c1170d5cb783868b125c0d5e66c654580db7621b46e9dd0cfa2f727d5ec4ab7
Writing manifest to image destination
Storing signatures
2023/01/06 15:51:32  info unpack layer: sha256:11323ed2c65349758e68a03a8e43825ec263dc9790daea93cf83b18ad0703109
2023/01/06 15:51:33  info unpack layer: sha256:0aaee4d952f0088f1c397964693a99ed7a8e809a7f087a2456c101e7d3bd5df5
2023/01/06 15:51:33  info unpack layer: sha256:003bcea46c04fbe5893ee7c6bc276fe337f2ac00f34c75501c5f4871ed26c7a1
2023/01/06 15:51:33  info unpack layer: sha256:08f9fb368cf50d6fee11f05a3683312e842188c4660f72b98e8b728ae483b63e
2023/01/06 15:51:33  info unpack layer: sha256:b4c05861b4dfa58a080ee2454b61d3caf1d1320eea61e0a655d51739019cae2a
2023/01/06 15:51:33  info unpack layer: sha256:3d07ae8a5cfae06e8b59657be91de80f8b1756cac34f33093164d5b6f27b7a60
2023/01/06 15:51:40  info unpack layer: sha256:93e0937bce3de61746cd07828ac14bbfd6be2543b61c2cb36591693a806f6310
2023/01/06 15:51:40  info unpack layer: sha256:a818ff5e3d218ac3e88f5f7d5492e35cdb9d041978d004463f9d2639670be6b8
2023/01/06 15:51:52  info unpack layer: sha256:7e4a2bed2378dd30858a12c1b12d05ef404139edc37e3a6a4cbb491e6ccf5039
2023/01/06 15:51:52  info unpack layer: sha256:af3ca23b3e56f41b70530e7f13e333b5c412e5c361bc55f587ddcde5d2bf8ac5
2023/01/06 15:52:00  info unpack layer: sha256:8fe3bf88b8a2ead72c89d699a1f055f55a8aa108b611e5366834546fc589baa4
2023/01/06 15:52:00  info unpack layer: sha256:7d94343897f22a68f039f283f56c631d3d8840aea350b3ae18bff1174f85b087
2023/01/06 15:52:00  info unpack layer: sha256:09949d2afee007e3aa3892a4afd94ed8a9b7e848ebb6fddd62a08a52b52faec2
2023/01/06 15:52:00  info unpack layer: sha256:e3f6e4676372867d91234c8fd1ac0a7dd3c45f4ef7230943ec4419f7108f98ac
2023/01/06 15:52:00  info unpack layer: sha256:c1e66fd9086256245bc314a9c49c3ccea2584069cbaf5202d6705c61f4107a7d
2023/01/06 15:52:00  info unpack layer: sha256:00f9fc2ec83e80c4973839ce6a6b3662fecbc32f331266ed26a700ad2c828694
2023/01/06 15:52:00  info unpack layer: sha256:d1ff7f4ba7b321e913ef7f23f42f151b0dedd7c1e256df179d11d832e0f55338
2023/01/06 15:52:00  info unpack layer: sha256:04a6139bd10d52ef01dd3feb018ddc46522878c47e80c7fad60a4d80a9890dd9
2023/01/06 15:52:02  info unpack layer: sha256:bcfbc387c0656eba0cdb459dfd30daec7570465ebecce4d5ddebb0f0972c7c7c
2023/01/06 15:52:06  info unpack layer: sha256:4cdecdba6820e97126f6e9e512d20da229cdaee5d5942ac0643dbdd7d1fc561d
2023/01/06 15:52:06  info unpack layer: sha256:71e92c1ce896438c6d4de350fce4d7ac7e6234f96ee297d056daa42156f5c671
2023/01/06 15:52:06  info unpack layer: sha256:35458b947e57b18e0c39801519885f99ca6a55ec01eb83452f438e531460107b
2023/01/06 15:52:34  info unpack layer: sha256:5a9cb6c9cef30b98f8ad331e22dad67eb7d9297967b699221527ff1b6e8c4f84
2023/01/06 15:52:39  info unpack layer: sha256:322f61cc85dfc15de2bdca3dfd60b88e01e883527ac8fa3095584e45051ff433
INFO:    Creating SIF file...
yass-base:latest.sif
WARNING: underlay of /etc/localtime required more than 50 (97) bind mounts
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (348) bind mounts

Traceback (most recent call last):
  File "/home/rth/spikeiterface/theprocess.py", line 110, in <module>
    yass_sorting = si.run_yass(recording=recording_saved, output_folder="yass",singularity_image="spikeinterface/yass-base:latest")
  File "/home/rth/.local/apps/spikes/lib/python3.10/site-packages/spikeinterface/sorters/runsorter.py", line 730, in run_yass
    return run_sorter('yass', *args, **kwargs)
  File "/home/rth/.local/apps/spikes/lib/python3.10/site-packages/spikeinterface/sorters/runsorter.py", line 137, in run_sorter
    return run_sorter_container(
  File "/home/rth/.local/apps/spikes/lib/python3.10/site-packages/spikeinterface/sorters/runsorter.py", line 578, in run_sorter_container
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting in singularity failed with the following error:
Traceback (most recent call last):
  File "/home/rth/spikeiterface/in_container_sorter_script.py", line 3, in <module>
    from spikeinterface import load_extractor
ModuleNotFoundError: No module named 'spikeinterface'

I can confirm that the same happens if I run the container in a shell

singularity shell yass-base\:latest.sif 
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (97) bind mounts
Singularity> python
Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import spikeinterface
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'spikeinterface'

Spikeubterface isn't installed, it seems.

Singularity> ls /home/miniconda3/lib/python3.7/site-packages
Cerberus-1.3.4.dist-info                       conda-4.13.0-py3.7.egg-info             joblib-1.1.0.dist-info            pkg_resources                                  six.py
Cython                                         conda_env                               kiwisolver                        progressbar                                    sklearn
Cython-0.29.32.dist-info                       conda_package_handling                  kiwisolver-1.4.4.dist-info        progressbar2-4.0.0.dist-info                   socks.py
OpenSSL                                        conda_package_handling-1.6.0.dist-info  matplotlib                        pyOpenSSL-19.1.0.dist-info                     sockshandler.py
PIL                                            cryptography                            matplotlib-3.5.2-py3.7-nspkg.pth  pycosat-0.6.3-py3.7.egg-info                   statsmodels
Pillow-9.2.0.dist-info                         cryptography-2.8.dist-info              matplotlib-3.5.2.dist-info        pycosat.cpython-37m-x86_64-linux-gnu.so        statsmodels-0.13.2.dist-info
Pillow.libs                                    cycler-0.11.0.dist-info                 mkl                               pycparser                                      test_pycosat.py
PySocks-1.7.1.dist-info                        cycler.py                               mkl_fft                           pycparser-2.19-py3.7.egg-info                  threadpoolctl-3.1.0.dist-info
PyYAML-5.3.dist-info                           cython.py                               mkl_fft-1.3.0-py3.7.egg-info      pylab.py                                       threadpoolctl.py
README.txt                                     dateutil                                mkl_random                        pyparsing                                      torch
__pycache__                                    deconv-0.0.0-py3.7-linux-x86_64.egg     mkl_random-1.1.1.dist-info        pyparsing-3.0.9.dist-info                      torch-1.2.0.dist-info
_cffi_backend.cpython-37m-x86_64-linux-gnu.so  dill                                    mkl_service-2.3.0-py3.7.egg-info  python_dateutil-2.8.2.dist-info                tqdm
_multiprocess                                  dill-0.3.5.1.dist-info                  mpl_toolkits                      python_utils                                   tqdm-4.42.1.dist-info
asn1crypto                                     diptest                                 multiprocess                      python_utils-3.3.3.dist-info                   typing_extensions-4.3.0.dist-info
asn1crypto-1.3.0.dist-info                     easy-install.pth                        multiprocess-0.70.13.dist-info    pytz                                           typing_extensions.py
caffe2                                         easy_install.py                         networkx                          pytz-2022.1.dist-info                          urllib3
cerberus                                       fontTools                               networkx-2.6.3.dist-info          pyximport                                      urllib3-1.25.8.dist-info
certifi                                        fonttools-4.34.4.dist-info              numpy                             requests                                       wheel
certifi-2022.6.15.dist-info                    h5py                                    numpy-1.19.2.dist-info            requests-2.22.0.dist-info                      wheel-0.34.2-py3.7.egg-info
cffi                                           h5py-3.7.0.dist-info                    packaging                         ruamel_yaml                                    xontrib
cffi-1.14.0.dist-info                          h5py.libs                               packaging-21.3.dist-info          ruamel_yaml-0.15.87-py3.7.egg-info             yaml
chardet                                        humanfriendly                           pandas                            scikit_learn-1.0.2.dist-info                   yass
chardet-3.0.4.dist-info                        humanfriendly-10.0.dist-info            pandas-1.3.5.dist-info            scikit_learn.libs                              yass_algorithm-2.0.dist-info
click                                          idna                                    parmap                            scipy                                          zipp-3.8.1.dist-info
click-8.1.3.dist-info                          idna-2.8-py3.7.egg-info                 parmap-1.5.3.dist-info            scipy-1.7.3.dist-info                          zipp.py
coloredlogs                                    importlib_metadata                      patsy                             scipy.libs
coloredlogs-15.0.1.dist-info                   importlib_metadata-4.12.0.dist-info     patsy-0.5.2.dist-info             setuptools
coloredlogs.pth                                inplace-0.0.0-py3.7-linux-x86_64.egg    pip                               setuptools-45.2.0.post20200210-py3.7.egg-info
conda                                          joblib                                  pip-20.0.2-py3.7.egg-info         six-1.14.0.dist-info

Any idea why is that and how to fix it?

[alejoe91]

Hi @rat-h

We need to investigate this, but, in principle, spikeinterface is installed at run-time in the container here: https://github.com/SpikeInterface/spikeinterface/blob/master/spikeinterface/sorters/runsorter.py#L488-L518

Can you try to run with verbose=True? I'll test on my side

[alejoe91]

@rat-h sorry for my late reply on this. Unfortunately, YASS requires py3.7 and as SpikeInterface version 0.96 we only support python>=3.8. This makes YASS legacy, as noted here: https://spikeinterface.readthedocs.io/en/latest/install_sorters.html#yass-legacy