Closed ethanbb closed 1 month ago
Changed to draft because I'm actually still refactoring things as part of the cross-platform saving fix
tox-dev/filelock#331 is currently a blocker... could probably sidestep but I'm hoping they fix it quickly (for now)
This now also fixes #209 - sorry for the lack of clean separation between them. Locking the batch file while writing was involved in both sets of changes.
I made some significant changes to put the saving functionality in the caiman
dataframe extension and streamline locking the file where necessary - please take another look when you can!
Hmm checks seem to be failing at the install step - not sure if it's due to these changes or something else is going on.
have a few comments, sorry been busy with a big fastplotlib
release. You can always always ping me with @ if I forget
No worries, I've been working on other things as well.
@kushalkolar done making changes for now; see remaining threads for questions I still have
This implements the "slurm" backend via
_run_slurm
. Supports passing a partition or list of partitions on which to run the jobs; more options (such as memory allocation) could be added if we think they're important.For controlling number of CPUs/processes per job, right now I'm using the MESMERIZE_N_PROCESSES environment variable to indicate how many processes to use per job, which matches the behavior of the "subprocess" backend. However, it might be a good idea to instead divide this number by the number of jobs that are running in parallel, to avoid needlessly over-parallelizing each job. I'm doing this in my own code to set the environment variable and it works fine, but it may make sense to automate it.
This also adds a dependency on
filelock
to lock the batch file when updating it to avoid race conditions. I used theSoftFileLock
because the regularFileLock
wasn't working for me on a NTFS remote drive (from Linux). As I understand it, this basically just creates a lock file when acquiring and deletes it when releasing, which also wouldn't be hard to implement ourselves if you prefer not to add a dependency.