search

SpikeInterface / spikeinterface

A Python-based module for creating flexible and robust spike sorting pipelines.
https://spikeinterface.readthedocs.io
MIT License
527 stars 187 forks source link

non-rigid motion correction on 2hr NPX recording #2477

Open atombysx opened 9 months ago

atombysx commented 9 months ago

Hi, I'm testing spikeinterface on concatenating chronic NPX 1.0 recordings on at least two days and then spike sort.

I'm trying to use the non-rigid method motion correction which has been running smoothly on my one day recordings - about 1-1.5hrs.

After concatenating two days' recordings, I have:

all concatenated ConcatenateSegmentRecording: 383 channels - 30.0kHz - 1 segments - 288,029,739 samples 9,600.99s (2.67 hours) - int16 dtype - 205.48 GiB

The detect and localize peaks step runs okay, but the python session is killed after:

detect and localize: 100%|██████████████████████████████| 9601/9601 [5:01:00<00:00, 1.88s/it] Killed

I wonder if this is due to memory usage? I'm running the session on Ubuntu 22.04, 64GB RAM, 8 Cores Intel Xeon E5-2640, Nvidia Titan X.

alejoe91 commented 9 months ago

Hi @atombysx

Can you upgrade spikeinterface to 0.100.0? There was a hidden bug that caused the inference part of the motion to use an enormous amount of RAM, and I think that is killing your process. Should be fixed in the new version!

See: https://github.com/SpikeInterface/spikeinterface/pull/2263

atombysx commented 9 months ago

Hi @atombysx

Can you upgrade spikeinterface to 0.100.0? There was a hidden bug that caused the inference part of the motion to use an enormous amount of RAM, and I think that is killing your process. Should be fixed in the new version!

See: #2263

Hi I just re-ran the same lines after upgrading.

The same happened: all concatenated ConcatenateSegmentRecording: 383 channels - 30.0kHz - 1 segments - 288,029,739 samples 9,600.99s (2.67 hours) - int16 dtype - 205.48 GiB detect and localize: 100%|██████████████████████████████| 9601/9601 [4:57:46<00:00, 1.86s/it] Killed

I upgraded by pip install --upgrade spikeinterface

The following info for pip show spikeinterface: Name: spikeinterface Version: 0.100.0 Summary: Python toolkit for analysis, visualization, and comparison of spike sorting output Home-page: Author: Author-email: Alessio Buccino <alessiop.buccino@gmail.com>, Samuel Garcia <sam.garcia.die@gmail.com> License: Location: /home/lab/.conda/envs/allenPipelineEnv/lib/python3.8/site-packages Requires: neo, numpy, probeinterface, threadpoolctl, tqdm, zarr Required-by: