search

flatironinstitute / CaImAn

Computational toolbox for large scale Calcium Imaging Analysis, including movie handling, motion correction, source extraction, spike deconvolution and result visualization.
https://caiman.readthedocs.io
GNU General Public License v2.0
639 stars 370 forks source link

Tracking: slurm cleanup #1213

Closed pgunn closed 1 year ago

pgunn commented 1 year ago

The code for the slurm backend to setup_cluster() is embarrassingly bad, with hardcoded paths. It also probably doesn't work (the codebase has a lot of explicit mention of python2.x).

We should either remove that code or clean it up. Urgently.

pgunn commented 1 year ago

Made a small (untested, but might work) effort towards fixing this in #1214

pgunn commented 1 year ago

The SLURM/ directory still needs a lot of attention, if we think it's worth even fixing this code. I'm pretty sure some of those imports won't work anymore. Even after fixing them, we'd need to figure out and document a lot of this stuff to give people a fighting chance.

EricThomson commented 1 year ago

This would be really helpful IMO. There are people I've talked to offline, at workshops, who roll their own solution. They tend to be power users and not ask for help and sort of just figure it out on their own (they often have HPC people helping). But there are also probably a lot of people who would appreciate a nudge in the right direction with a little example shell script or two.

pgunn commented 1 year ago

setup_cluster() is adjusted (since 1214 landed) and now could at least in theory work.

Next steps to satisfy this tracking issue:

1) Cleanup SLURM/ 2) Test all of this 3) Document all of this