search

LorenFrankLab / spyglass

Neuroscience data analysis framework for reproducible research built by Loren Frank Lab at UCSF
https://lorenfranklab.github.io/spyglass/
MIT License
91 stars 41 forks source link

Clean up /recording more automatically? #183

Open acomrie opened 2 years ago

acomrie commented 2 years ago

It appears that there are a number of old recording directories in /stelmo/nwb/recording/ corresponding to entries in an old version of the SpikeSortingRecording table that has since been dropped. If there isn't already a method that cleans these up, it might be nice to add.

Related to this - I interrupted a populate() call on SpikeSortingRecording, and upon re-running later, I got an error that the directory corresponding to that table entry already existed in /stelmo/nwb/recording/. Rather than the code reusing or overwriting the existing dir, I had to delete that empty dir manually. This was fine of course, but it could be nice if there was some clean-up functionality here too. For ex, so that directories and files are cleaned up from stelmo in cases where final insertion into the table failed for whatever reason.

None of this is "blocking" but would be nice to avoid building up unnecessary data on stelmo if possible. Thanks.

lfrank commented 2 years ago

Agreed. We can write clean up code that is similar to the cleanup for the nwb / analysisnwb files where it finds the directories that are not in the table and removes them. In the meantime I'd be happy for us to empty those directories completely, but I have a sense that a few people might still be using stuff there.

khl02007 commented 9 months ago

Note that if you use spike sorting v1 pipeline, the spike sorting gets saved in the NWB format (as analysis NWB files) so we no longer need a separate /recording dir. Having said that, it's still probably a good idea to have a function that deletes recordings not in v0 SpikeSortingRecording from the /recording dir