Test containers with Apptainer

[alejoe91]

All sorters have been tested with Singularity. We should also test wiht Apptainer since this will be the long-term supported version

[ghost]

I've tested a few different sorters with apptainer version 1.1.7-1.el8.

Note as I mentioned in a previous thread, the containers can't install anything once they've been built, so I'm therefore building them from definition files like:

Bootstrap: docker
From: spikeinterface/mountainsort5-base:latest

%post
pip install git+https://github.com/SpikeInterface/spikeinterface.git@main

With the appropriate image and version of spikeinterface (ie 0.95.1 for klusta, latest dev for mountainsort5, 0.97.1 for everything else). This may well be only necessary because of the configuration of apptainer on my system.

My results from running these containers on a 5 minute tetrode recording with default parameters are:

• klusta: good
• ironclust: good
• pykilosort: good
• mountainsort4: good
• mountainsort5: good
• kilosort3: failing
• kilosort2_5: failing

I'm not sure why both kilosorts are failing as pykilosort is fine, so it can't be a CUDA issue, and ironclust is working, so it's not a matlab problem either. I also had to manually install scipy into the kilosort3 container as it wasn't available if i just installed spikeinterface.

The error message for kilosort3 is:

SpikeSortingError: Spike sorting in singularity failed with the following error:
['RUNNING SHELL SCRIPT: /lmb/home/rbedford/Documents/testsort/apptainer/kilosort3/sorter_output/run_kilosort3.sh\nTime   0s. Computing whitening matrix.. \n\nGetting channel whitening matrix... \n\nChannel-whitening matrix computed. \n\nTime  11s. Loading raw data and applying filters... \n\nTime  15s. Finished preprocessing 220 batches. \n\nDrift correction ENABLED\n\nvertical pitch size is 1.000000e+01 \n\nhorizontal pitch size is 5 \n\n  Columns 1 through 7\n\n\n\n  -10.0000   -7.5000   -5.0000   -2.5000         0    2.5000    5.0000\n\n\n\n  Columns 8 through 9\n\n\n\n    7.5000   10.0000\n\n\n\n     6\n\n\n\n----------------------------------------Index in position 1 exceeds array bounds. Index must not exceed 4.\nError running kilosort3\nTraceback (most recent call last):\n  File "/home/miniconda3/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 237, in run_from_folder\n    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n  File "/home/miniconda3/lib/python3.8/site-packages/spikeinterface/sorters/external/kilosortbase.py", line 209, in _run_from_folder\n    raise Exception(f"{cls.sorter_name} returned a non-zero exit code")\nException: kilosort3 returned a non-zero exit code\n\n', '\rwrite_binary_recording:   0%|          | 0/301 [00:00<?, ?it/s]\rwrite_binary_recording:   2%|▏         | 5/301 [00:00<00:05, 49.44it/s]\rwrite_binary_recording:   3%|▎         | 8/301 [00:00<00:07, 41.11it/s]\rwrite_binary_recording:  31%|███       | 92/301 [00:00<00:03, 57.41it/s]\rwrite_binary_recording: 100%|██████████| 301/301 [00:00<00:00, 946.10it/s]\nTraceback (most recent call last):\n  File "/lmb/home/rbedford/Documents/testsort/apptainer/in_container_sorter_script.py", line 17, in <module>\n    sorting = run_sorter_local(\n  File "/home/miniconda3/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py", line 167, in run_sorter_local\n    SorterClass.run_from_folder(output_folder, raise_error, verbose)\n  File "/home/miniconda3/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 273, in run_from_folder\n    raise SpikeSortingError(\nspikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting failed. You can inspect the runtime trace in /lmb/home/rbedford/Documents/testsort/apptainer/kilosort3/spikeinterface_log.json\n']

And kilosort2_5 has basically the exact same indexing error.

I'm not sure if this is an apptainer specific problem or something else. It would be nice to get kilosort3 running. It's fantastic that all the other sorters I've tested are working though! Let me know if it would be helpful to complete this list and test all the others. I haven't done that because I don't think we'll have a need for any sorters not on this list but it would be easy to do.

[TsungChihTsai]

I tried to use all list of docker when I use open ephys recording system to save binary format and the probe is 'ASSY-196-P-1' from 'cambridgeneurotech', I found that

pykilosort: good kilosort: good Kilosort2: good mountainsort4: good kilosort3: good, but in a dataset (from se.read_ibl_streaming_recording('ae8787b1-4229-4d56-b0c2-566b61a25b77',stream_name='probe00.ap')) is failing ("returned a non-zero exit code") yass: failing Klusta: failing herdingspikes: failing tridesclous: failing ( only 1 to 2 units) waveclus_snippets: failing Hdsort: failing

Problem is "returned a non-zero exit code" as below: kilosort2_5: failing Spykingcircus2: failing Ironclust: failing Waveclus: failing spykingcircus: failing

Now, I'm just wondering why kilosort3 sometime is workable, sometime showed "returned a non-zero exit code" ?

[zm711]

@alejoe91 is this good now or not? :)

[alejoe91]

I think we can close. I plan to revive the automated singularity tests but it will take some time :)

[zm711]

Cool sounds good. I think a new issue will be easier for us to find :)