Kilosort2 and IronClust "return a non-zero exit code" when run on different recordings

[cristofer-holobetz]

I am attempting to sort multiple recordings with Kilosort2 and IronClust. For some recordings, both sorters work. For others, one or the other does not.

For the recording from animal RN44, Kilosort2 fails but IronClust works. For the recording from animal CH61, IronClust fails but Kilosort2 works.

Here are the error traces for two such sorting attempts. Both failures have the same vague form:

Kilosort2:

SpikeSortingError: Spike sorting in singularity failed with the following error:
['Traceback (most recent call last):\n  File "/data/RN44/saved_sortings/KS2_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 237, in run_from_folder\n    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n  File "/data/RN44/saved_sortings/KS2_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/external/kilosortbase.py", line 194, in _run_from_folder\n    raise Exception(f"{cls.sorter_name} returned a non-zero exit code")\nException: kilosort2 returned a non-zero exit code\n\n', 'Traceback (most recent call last):\n  File "/data/RN44/saved_sortings/KS2_sortings/in_container_sorter_script.py", line 17, in <module>\n    sorting = run_sorter_local(\n  File "/data/RN44/saved_sortings/KS2_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py", line 167, in run_sorter_local\n    SorterClass.run_from_folder(output_folder, raise_error, verbose)\n  File "/data/RN44/saved_sortings/KS2_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 273, in run_from_folder\n    raise SpikeSortingError(\nspikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting failed. You can inspect the runtime trace in /data/RN44/saved_sortings/KS2_sortings/Alternation/spikeinterface_log.json\n']

IronClust:

SpikeSortingError: Spike sorting in singularity failed with the following error:
['Traceback (most recent call last):\n  File "/data/CH61/saved_sortings/IC_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 237, in run_from_folder\n    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n  File "/data/CH61/saved_sortings/IC_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/external/ironclust.py", line 264, in _run_from_folder\n    raise Exception("ironclust returned a non-zero exit code")\nException: ironclust returned a non-zero exit code\n\n', 'Traceback (most recent call last):\n  File "/data/CH61/saved_sortings/IC_sortings/in_container_sorter_script.py", line 17, in <module>\n    sorting = run_sorter_local(\n  File "/data/CH61/saved_sortings/IC_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py", line 167, in run_sorter_local\n    SorterClass.run_from_folder(output_folder, raise_error, verbose)\n  File "/data/CH61/saved_sortings/IC_sortings/in_container_python_base/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 273, in run_from_folder\n    raise SpikeSortingError(\nspikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting failed. You can inspect the runtime trace in /data/CH61/saved_sortings/IC_sortings/Exploration/spikeinterface_log.json\n']

And here are the respective logs:

Kilosort2: spikeinterface_log .txt

IronClust: spikeinterface_log .txt

Here is the code that generated these exceptions:

for animal_id in all_animals_all_days_paths.keys():

    for behavior_day in ['Exploration', 'Alternation']:

        BIN_RECORDING = REMOVE_BAD_CHANNELS_DICT[animal_id][behavior_day]
        logging.info('Recording for {0} {1}: {2}'.format(animal_id, behavior_day, BIN_RECORDING))

        # KiloSort2
        time0 = time()

        logging.info('KiloSort2 for {0} {1} started'.format(animal_id, behavior_day))

        KS2_SORTING = ss.run_sorter('kilosort2',
                                    BIN_RECORDING,
                                    output_folder='/data/{0}/saved_sortings/KS2_sortings/{1}'.format(animal_id, behavior_day),
                                    singularity_image="spikeinterface/kilosort2-compiled-base:latest")
        time1 = time()

        logging.info('KiloSort2 for {0} {1} finished'.format(animal_id, behavior_day))
        logging.info('KiloSort2 for {0} {1} took {2} hours\n'.format(animal_id, behavior_day, (time1 - time0) / (60 * 60)))

        KS2_SORTING.dump_to_json('/data/{0}/saved_sortings/KS2_sortings/KS2_{1}.json'.format(animal_id, behavior_day))

        #IronClust
        time0 = time()

        logging.info('IronClust for {0} {1} started'.format(animal_id, behavior_day))

        IC_SORTING = ss.run_sorter('ironclust',
                                    BIN_RECORDING,
                                    output_folder='/data/{0}/saved_sortings/IC_sortings/{1}'.format(animal_id, behavior_day),
                                    singularity_image="spikeinterface/ironclust-compiled-base:latest")

        time1 = time()

        logging.info('IronClust for {0} {1} finished'.format(animal_id, behavior_day))
        logging.info('IronClust for {0} {1} took {2} hours\n\n'.format(animal_id, behavior_day, (time1 - time0) / (60 * 60)))

        IC_SORTING.dump_to_json('/data/{0}/saved_sortings/IC_sortings/IC_{1}.json'.format(animal_id, behavior_day))

The fact that the sorters fail on different recordings makes me think that it is not a problem with the recordings themselves, but I am not confident this is the case.

[alejoe91]

Hi @cristofer-holobetz

Unfortunately sometimes sorters fail for sorter-specific reasons :(

How many units do you get for both recordings? Have you checked if waveforms etc. look allright?

[cristofer-holobetz]

Hi @alejoe91!

I've been traveling lately. I'll take a closer look at the details of the recordings that you mentioned. Thanks so much for your help so far though :)