Docker image for spike sorting of nwb files

[magland]

Toward the goal of running spike sorting on some data in docker container

• [ ] Make a spikeforest datalad dataset so that we have prelim data to work with (starting with paired_kampff): https://github.com/spikodrome/datasets/issues/3
• [ ] Convert the .mda file structure to .nwb using spikeinterface. Here's the relevant pull request from @bendichter: https://github.com/SpikeInterface/spikeextractors/pull/285
• [ ] Make an outside-of-docker spike sorting script (we'll start with mountainsort) that is capable of handling .nwb files for I/O (using spikeinterface) - and that includes some visualizations so that we can verify it is working
• [ ] Dockerize that script using the dockerfile from spikeforest as a basis
• [ ] Add more sorters -- we'll try to get them all into one docker image, and we'll handle conflicts as they arise
• [ ] Think about supporting the MATLAB sorters -- there are various ways forward here, and we need to consider gpu.

@yarikoptic @bendichter

[magland]

Revised plan, since I decided to keep the SF data in original spikeforest format rather than converting to nwb. This is partly because .nwb spikeinterface wrapper PR is not integrated yet. The other part is that the mda format is a lot simpler.

• [x] Make a spikeforest recordings repo so we have prelim data to work with: see https://github.com/flatironinstitute/spikeforest_recordings
• [x] Make an AutoRecordingExtractor capable of handling many recording input formats - including mda and ultimately will also support nwb. See: https://github.com/flatironinstitute/ephys-viz/tree/master/widgets/pycommon/autoextractors
• [x] Make an outside-of-docker spike sorting script that utilizes the AutoRecordingExtractor and produces a directory of outputs -- including sorting results and other data - metrics, etc
• [x] Dockerize that script using the dockerfile from spikeforest as a basis
• [x] Add more sorters -- we'll try to get them all into one docker image, and we'll handle conflicts as they arise
• [x] Think about supporting the MATLAB sorters -- there are various ways forward here, and we need to consider gpu.

[yarikoptic]

This is partly because .nwb spikeinterface wrapper PR is not integrated yet.

The plan sounds good to me -- thanks for keeping the project going forward! I hope that .nwb interface would be finalized soon, and then SF / the docker image would simply acquire additional super powers of being able to consume/produce .nwb files. I will soon stop traveling, and hope to finally contribute more than just words here and there ;-)

[magland]

Finished the above checklist -- prototypes only. And the docker containers are here: https://github.com/flatironinstitute/spikeforest/tree/master/future_containers

The most exciting thing is that we (James and I) managed to use matlab compiler to dockerize ironclust and kilosort2

@yarikoptic

[yarikoptic]

@magland

• it seems that you are aiming for a collection of docker images (one per sorting algorithm) or eventually just a single one?
• did you push pre-generated images to any hub already?

[yarikoptic]

did you push pre-generated images to any hub already?

silly me now realized -- MATLAB! but I wonder if it could still be legit for redistribution, after all license is not redistributed and could be provided "at run time" by the user.

[magland]

It's perfectly fine and legal to distribute compiled MATLAB programs -- but the person generating the binaries needs to have a license for matlab compiler.

Yes, images have been pushed to dockerhub.

For an example, see: https://github.com/flatironinstitute/spikeforest/blob/master/future_containers/sorters/spykingcircus/test1_docker.sh

The call is a bit complex... but that's partly because I'm using kachery to auto-download the needed recording files. (pip install kachery)

[magland]

First pass is one container per sorter (because that's hard enough). At some point we can think about one bit container. But there are many disadvantages when things are in flux.