search

m-beau / NeuroPyxels

NeuroPyxels (npyx) is a python library built for electrophysiologists using Neuropixels electrodes. This package stems from the need of a pythonist who really did not want to transition to MATLAB to work with Neuropixels: it features a suite of core utility functions for loading, processing and plotting Neuropixels data.
GNU General Public License v3.0
109 stars 26 forks source link

The size of .Neuropyxels folder causes disc size problems on the server #392

Closed sevgiozturk closed 1 week ago

sevgiozturk commented 3 months ago

Hi Maxime, for C4 version of npyx, we are having problems with expanding .Neuropixels folder for each user on the server. It is growing up to 500 GB for a recording of 90 GB despite deleting that folder between recordings. There could be two solutions I can think of: Either we should be able to direct this folder under other drives - not under /home directory or nypx can delete it just after it finishes processing unless it needs it to reopen the same recording.

Can you help us solve this issue? We're running into the same issue multiple times in a week.

Thanks Sevgi - Hull lab

m-beau commented 3 months ago

Hi Sevgi,

Yes I was aware that this was a problem (see also #384), and finally got around finding a futureproof solution over the past week: https://github.com/m-beau/cachecache

I couldn't find the right tool out there so had to write it from scratch. This will ensure that the cached data is distributed across datasets, where the data is sitting instead of the C drive. It also implements a limit to the cache size, depending on the available memory on disk.

It is working on my own branch, but I want to keep testing it for a couple of weeks in the real world before pushing it to the master branch. In the mean time... I suggest that you just delete the .NeuroPyxels folder when it gets too big. Sorry about that!

Maxime

m-beau commented 1 month ago

Hi Sevgi,

It seems like the new caching mechanism is stable. Cached files are now saved at dp/.NeuroPyxels by default, and it is also possible to enforce a specific directory for caching using the 'cache_path' argument all over the place.

Can you download and use the m-beau branch of Neuropixels and let me know whether it solves your issues? Feel free to email me.

sevgiozturk commented 1 week ago

Hi Maxime, thanks for the fix. Now, I'm able to see it is creating that folder under dp. However, I still see .Neuropyxels folder is created under my home (see the top-left folder structure in the attachment). Do you think I should keep deleting this folder while switching between recordings so that it doesn't confuse cached data among the recordings?

PhyllumUpdated
m-beau commented 1 week ago

Hey,

Do not worry, this is normal - normally, there should also be a /.NeuroPyxels folder in the local directory with your recording, right? This is where most of the cached files should be saved.

sevgiozturk commented 1 week ago

Yes, the one under the recording folder is no problem. My question is more about the one still created under home (top left in the figure). Should I delete this folder when I'm done curating one recording with Phyllum and switching to another recording? Since we adopted deleting common .Neuropyxels folder under home between switching to other recordings so that it does not load previous cached files for the new recording, I wonder if I should keep deleting it since it is still created.

m-beau commented 1 week ago

No, you do not need to delete it - there shouldn’t be any crosstalk (and even it the past, it should have happened super rarely).

Let me know if it ever happens in the future, because it means there is a bug I need to fix!

sevgiozturk commented 1 week ago

Ok, I hope it doesn't. I'm closing this issue since the original bug is fixed. Thanks Maxime!