mountainsort5 returns zero units

[RikkelBob]

I'm trying to compare mountainsort4 and 5. However, mountainsort5 returns zero units (where mountainsort4 returned ~80 units). It seems to run fine in docker, but after a few minutes it quits and acts like it has finished. I was thinking some error could be happening in docker, but asking for the logs of the container just returns the python version.

Any idea on how to proceed?

[alejoe91]

Mmm can you try to run it "locally"? MS5 is very easy and lightweight to install: pip install mountainsort5

what is your input data? MS5 has different sorting "schemes". Maybe you're not using the good one?

@magland any idea?

[magland]

Yes, I agree with @alejoe91, please try running it outside the container. Please provide the console output (or at least the last part of it) for the mountainsort5 run, so we can see what the problem might be.

[florian6973]

Hi! I just tried to run it inside Docker on Windows and I got modulenotfounderror 'h5py', I do not know if it can help

[RikkelBob]

@alejoe91 @magland Thanks for the suggestions, running it outside of docker has the same results! I'm just sorting a snippet, so that could be the cause. I'll run a full session overnight and report the results

@florian6973 This can be resolved by setting the environment variable SPIKEINTERFACE_DEV_PATH I believe

[alejoe91]

@florian6973 can you paste the full error?

[florian6973]

Thanks for the tip about SPIKEINTERFACE_DEV_PATH!

It seems to have been an error related to an unstable network, I was using a LTE hotspot and by trying again at home, it seems to work fine, with no errors and the expected units. Very sorry about this false positive, it is just a bit sad that it is hard to get the full log from Docker and the error messages are not always very precise. Except that, containers are very helpful in any case!

[alejoe91]

Can we close the issue then? :)

[RikkelBob]

Well, I'm getting 1 unit now when sorting the full recording instead of 0 (I'm positive this is not representative). This is not a spikeinterface issue though, so feel free to close :)

[florian6973]

@florian6973 can you paste the full error?

Just to follow up, I got the error today again (on a home/office network):

Running sorting
Available sorting methods: ['combinato', 'hdsort', 'herdingspikes', 'ironclust', 'kilosort', 'kilosort2', 'kilosort2_5', 'kilosort3', 'klusta', 'mountainsort4', 'mountainsort5', 'pykilosort', 'spykingcircus', 'spykingcircus2', 'tridesclous', 'tridesclous2', 'waveclus', 'waveclus_snippets', 'yass']
Running sorter mountainsort5 with params {'scheme': '2', 'detect_threshold': 5.5, 'detect_sign': -1, 'detect_time_radius_msec': 0.5, 'snippet_T1': 20, 'snippet_T2': 20, 'npca_per_channel': 3, 'npca_per_subdivision': 10, 'snippet_mask_radius': 250, 'scheme1_detect_channel_radius': 150, 'scheme2_phase1_detect_channel_radius': 200, 'scheme2_detect_channel_radius': 50, 'scheme2_max_num_snippets_per_training_batch': 200, 'scheme2_training_duration_sec': 300, 'scheme2_training_recording_sampling_mode': 'uniform', 'scheme3_block_duration_sec': 1800, 'freq_min': 300, 'freq_max': 6000, 'filter': True, 'whiten': True}
Starting container
Installing spikeinterface from sources in spikeinterface/mountainsort5-base
Installing dev spikeinterface from remote repository
Installing extra requirements: ['neo']
Running mountainsort5 sorter inside spikeinterface/mountainsort5-base
Stopping container
Unexpected error: Spike sorting in singularity failed with the following error:
Traceback (most recent call last):
  File "/cache/sorting/20a52e93ed40485fb35741fdda415d17/in_container_sorter_script.py", line 4, in <module>
    from spikeinterface.sorters import run_sorter_local
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/__init__.py", line 2, in <module>
    from .sorterlist import *
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/sorterlist.py", line 1, in <module>
    from .external.combinato import CombinatoSorter
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/external/combinato.py", line 10, in <module>
    from spikeinterface.extractors import CombinatoSortingExtractor
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/extractors/__init__.py", line 1, in <module>
    from .extractorlist import *
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/extractors/extractorlist.py", line 16, in <module>
    from .nwbextractors import (NwbRecordingExtractor, NwbSortingExtractor,
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/extractors/nwbextractors.py", line 5, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
 Traceback (most recent call last):
  File "/home/_code/pipeline/experiment.py", line 828, in spike_sorting
    sorting = ss.run_sorter(
  File "/home/.conda/envs/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 141, in run_sorter
    return run_sorter_container(
  File "/home/.conda/envs/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 594, in run_sorter_container
    raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting in singularity failed with the following error:
Traceback (most recent call last):
  File "/glia/cache/sorting/20a52e93ed40485fb35741fdda415d17/in_container_sorter_script.py", line 4, in <module>
    from spikeinterface.sorters import run_sorter_local
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/__init__.py", line 2, in <module>
    from .sorterlist import *
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/sorterlist.py", line 1, in <module>
    from .external.combinato import CombinatoSorter
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/external/combinato.py", line 10, in <module>
    from spikeinterface.extractors import CombinatoSortingExtractor
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/extractors/__init__.py", line 1, in <module>
    from .extractorlist import *
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/extractors/extractorlist.py", line 16, in <module>
    from .nwbextractors import (NwbRecordingExtractor, NwbSortingExtractor,
  File "/usr/local/lib/python3.8/site-packages/spikeinterface/extractors/nwbextractors.py", line 5, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'

Sorting failed - please fix it and retry

The code version is associated with the commit bca5051cffb911df4a80ae05cbd3c1fb12d3e947. When it did not happened the second time, I did not check the version.

[alejoe91]

This should be fixed in the main branch!

[florian6973]

Thanks for your feedback!

So in fact this error can happen after spikeinterface fails to be installed in the container (if I print the output result of the installation command). And indeed using the last commit of the main branch fixed it.

As a side note, if I do not update spikeinterface installed locally, I get a compability error after a few PRs are merged (related to relative_to for instance) with spikeinterface installed in the container. By any chance do you know if there is any way to make sure to install the code in the container with the same commit hash please? If not, I am going to stick with installing from a local folder to make sure this is the same version.

[alejoe91]

@florian6973 I made a PR with some explanation ;)

See #1786

[betty-hbr]

Well, I'm getting 1 unit now when sorting the full recording instead of 0 (I'm positive this is not representative). This is not a spikeinterface issue though, so feel free to close :)

Hi RikkelBob,

I wonder whether you have sorted out what causes the fact that MS5 detected few units than MS4. Same issues happened to me as well, 61 units were identified in MS5, but only 20~ were identified in MS5. Much appreciate if you could share some of your thoughts!

[magland]

Hi RikkelBob,

I wonder whether you have sorted out what causes the fact that MS5 detected few units than MS4. Same issues happened to me as well, 61 units were identified in MS5, but only 20~ were identified in MS5. Much appreciate if you could share some of your thoughts!

For reference I am linking to this issue https://github.com/flatironinstitute/mountainsort5/issues/44