Sorting fails, 'trigger' pipeline continues

bjhardcastle commented 8 months ago

A couple of recent sessions sorted with SpikeInterface==0.99.1 have failed with the same error message, and it's not clear to me why (see output below). I don't see the spikeinterface_log.json mentioned in the error message.

@jtyoung84 Unfortunately the trigger capsule continues despite the sorting capsule error and a data asset is created. The data are partially complete and it's not immediately obvious that sorting failed, so it's a bit annoying to have these data assets. They also contain a tmp folder with ~1 TB of raw data.

[ ] debug sorting error
[ ] re-sort sessions
[ ] delete assets below with large tmp folders

sorted assets

asset ID: 779eb3ca-9652-4476-a283-0022e2d3f1e4 asset ID: bd60d88c-b13d-4470-80d7-832360446947

example of `output`

...
Sorting recording: experiment1_Record Node 104#Neuropix-PXI-100.ProbeF-AP_recording1
BinaryFolderRecording: 362 channels - 30.0kHz - 1 segments - 233,969,572 samples 
                       7,798.99s (2.17 hours) - int16 dtype - 157.76 GiB
Traceback (most recent call last):
  File "run_capsule_aind.py", line 536, in <module>
    sorting = ss.run_sorter(
  File "/usr/local/lib/python3.8/dist-packages/spikeinterface/sorters/runsorter.py", line 147, in run_sorter
    return run_sorter_local(**common_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/spikeinterface/sorters/runsorter.py", line 173, in run_sorter_local
    SorterClass.run_from_folder(output_folder, raise_error, verbose)
  File "/usr/local/lib/python3.8/dist-packages/spikeinterface/sorters/basesorter.py", line 289, in run_from_folder
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/spikeinterface/sorters/basesorter.py", line 254, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/usr/local/lib/python3.8/dist-packages/spikeinterface/sorters/external/kilosortbase.py", line 215, in _run_from_folder
    raise Exception(f"{cls.sorter_name} returned a non-zero exit code")
Exception: kilosort2_5 returned a non-zero exit code

Spike sorting failed. You can inspect the runtime trace in /code/../scratch/spikesorted_raw/experiment1_Record Node 104#Neuropix-PXI-100.ProbeF-AP_recording1/spikeinterface_log.json.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_capsule_aind.py", line 547, in <module>
    shutil.copy(spikesorted_raw_output_folder / "spikeinterface_log.json", sorting_output_folder)
  File "/usr/lib/python3.8/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '../scratch/spikesorted_raw/spikeinterface_log.json'

alejoe91 commented 8 months ago

@bjhardcastle are you using the "stable" capsule or the "aind" capsule?

For the registration of wrong results, we might parse the output to check for "Errors". There is no direct way to get the exit status

bjhardcastle commented 8 months ago

This was with capsule on the aind branch.

AllenNeuralDynamics / aind-ephys-spikesort-kilosort25-full