Failure in run_sort_by_property

[thomas2cherian]

Hi,

I have implemented spikeinterface (with mountainsort4) on a dataset with large number of channels distributed across individual electrode arrays. I tested the sorting by splitting the recording based on the electrode array ID so that sorting runs only on the channels within each array. I then attempted to use the run_sorter_by_property function (to run the sorting across each unique array) as follows:

sortedData = si.run_sorter_by_property("mountainsort4", preprocessedData,
                                       grouping_property = 'fmaID',
                                       working_folder = baseFolder / "sorted",
                                       docker_image = True,
                                       verbose = True,
                                       detect_threshold = 3,
                                       filter = False,
                                       num_workers = 6,
                                       adjacency_radius = 50,
                                       clip_size = 50)`

The sorting seems to run fine (based on the verbose outputs) but I am encountering the error as follows (see errorLog.pdf attached for the full code). In the working folder, I see sub folders named with the arrayID that all only contain the folder 'mountainsort4' (with spikeinterface_job.json within).

SpikeSortingError: get result error. the folder do not contain spikeinterface_log.json

errorLog.pdf

Am I using the function incorrectly? Is there any alternative way to combine sorting output from split recordings?

[alejoe91]

Thanks for the report @thomas2cherian

Let me try to reproduce the issue!

[alejoe91]

@thomas2cherian

I was able to reproduce the issue and looking into it

[thomas2cherian]

@alejoe91 Thank you for looking into this issue!

[alejoe91]

@thomas2cherian can you test if this PR #1698 fixes your issue?

In brief you have to:

1. install SpikeInterface from that PR
2. set the following env variable: SPIKEINTERFACE_DEV_PATH="path-to-cloned-repo (this uses your local copy of SI in the container

Let me know if you have questions!

[thomas2cherian]

@alejoe91 Thank you for getting back. I will follow your suggestions, check if the fix works and get back to you. Just to be clear, I will be cloning the repository from the branch 'fix-caching-pre-container' listed in PR #1698.

[alejoe91]

yes, and install from the cloned repo with pip install -e .

[thomas2cherian]

Hi @alejoe91, I tried what you have suggested but unfortunately the issue persists. I am getting the following error now:

[Errno 2] No such file or directory: 'sorted\\6A\\mountainsort4\\spikeinterface_job.json'

The verbose output from run_sort_by_property are as follows:

Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container
Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container
Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container
Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container
Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container
Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container
Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container
Starting container
******
Container started with the following paths
/Users/pc/Downloads/spikeinterface-fix-caching-pre-container /sources
Installing spikeinterface from sources in spikeinterface/mountainsort4-base
Installing dev spikeinterface from local machine
Running mountainsort4 sorter inside spikeinterface/mountainsort4-base
Stopping container

As you can see above, the sorting has been attempted 8 times for each of the arrays used as grouping property. This output was provided in under a minute, which makes me doubt is sorting was actually even run (usually for one array, it takes ~500s on my system). Also this time the sub folders (named after each array ID) under sorter output folder are completely empty.

Please let me know if I should try something else.

[alejoe91]

So this means something is failing when running mountainsort. Strange because I was getting the same error and it's fixed in the PR. Would you mind sharing the data with me?

[thomas2cherian]

@alejoe91 Pardon me for getting back to you so late. We would be more than happy to share the data with you. Please give me a day (or two) to prepare a shared folder with all our codes and data which I will share this with you over email. Thank you!

[chimkasinma]

@alejoe91 @thomas2cherian hello! my team and I are also facing this challenge. We were wondering if there has been any update of some sorts. Thank you and hope to hear from you soon!

[thomas2cherian]

Hi @chimkasinma, @alejoe91 has prepared an update - PR #1698 that addresses this issue (I may have been incorporating this incorrectly on my local system, but it works on the same data file for @alejoe91). This update should be incorporated into the main spikeinterface branch within the coming week.

[alejoe91]

Should be fixed in >=0.98