search

MouseLand / Kilosort

Fast spike sorting with drift correction
https://kilosort.readthedocs.io/en/latest/
GNU General Public License v3.0
488 stars 247 forks source link

BUG: <Suddenly exited.> #794

Open Tan-Lao opened 1 month ago

Tan-Lao commented 1 month ago

Describe the issue:

Hi，everyone，

My previous data was fine, but this time the data always causes KiloSort 4 to suddenly exit during the drift_amount process.

The KiloSort 4 folder only contains an image of the drift_amount.

jacobpennington commented 1 month ago

Can you please attach a screenshot of what the Kilosort4 GUI looks like after loading your data, as well as kilosort4.log from the results directory?

Tan-Lao commented 1 month ago

Can you please attach a screenshot of what the Kilosort4 GUI looks like after loading your data, as well as kilosort4.log from the results directory?您能否附上加载数据后 Kilosort4 GUI 的屏幕截图，以及结果目录中的kilosort4.log？

Okay, the data import is normal, but it suddenly exits at the drift amount step. Below is the screenshot.

Bug Bug-kilosort4 loading data GUI

jacobpennington commented 1 month ago

Please try updating to the latest version (v4.0.18) and re-sorting, then upload the log file again (not just a screenshot, the actual file). The new version includes more details about resource usage in the log file, which may be related to the crash.

Tan-Lao commented 1 month ago

Ok, thanks for the answer. I'm going to update it and test it again.

Tan-Lao commented 1 month ago

Please try updating to the latest version (v4.0.18) and re-sorting, then upload the log file again (not just a screenshot, the actual file). The new version includes more details about resource usage in the log file, which may be related to the crash.

I have updated to kilosort4 (v4.0.18). I retested my data, and kilosort4 still exits abruptly after the drift amount. I also tested with kilosort3, using the same data, and kilosort3 works fine.

Attached below are the log files for kilosort4.

kilosort4.log

jacobpennington commented 1 month ago

Thanks. There's definitely something off, you're detecting a lot more spikes than would be expected for an hour-long Neuropixels recording and using a lot of extra memory as a result. My guess is it's crashing without an error because you're exceeding system memory at that step, it's hitting 90% usage or higher pretty regularly.

Two things to try: 1) Try changing all the values in probe['kcoords'] to the same number (all ones, all zeroes, doesn't matter). Other users have reported encountering problems with multi-shank probe layouts that resolve themselves if they treat the probe as a single shank. I'm still looking into what's causing that. 2) If it's feasible for you, can you share the data with me so that I can try to reproduce this for debugging? Whether or not (1) solves the problem, that would help me investigate the problem.

Tan-Lao commented 1 month ago

Thank you very much. I tried setting all the probe['kcoords'] parameters in the channel map to 1 and retested Kilosort4, which did not crash this time.

If my data can help you fix the bug, I would be more than happy to provide the raw data. The ap.bin file is about 83 GB. How can I send it to you?

jacobpennington commented 1 month ago

Google drive links have worked best so far, dropbox should be fine as well. You can paste the link here, or email it to me at @.***

On Thu, Oct 10, 2024, 9:14 PM Hulu Tan @.***> wrote:

Thank you very much. I tried setting all the probe['kcoords'] parameters in the channel map to 1 and retested Kilosort4, which did not crash this time.

If my data can help you fix the bug, I would be more than happy to provide the raw data. The ap.bin file is about 83 GB. How can I send it to you?

— Reply to this email directly, view it on GitHub https://github.com/MouseLand/Kilosort/issues/794#issuecomment-2406339004, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIQ6WYDTIT6SO2GG6IIKR2TZ24RATAVCNFSM6AAAAABPLMHW3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBWGMZTSMBQGQ . You are receiving this because you commented.Message ID: @.***>

Tan-Lao commented 1 month ago

Okay, I have uploaded the bin files and channel maps separately. Since I can’t see your email, I’ll put the links here. Below are their links.

https://www.dropbox.com/scl/fi/sv9xcw6n9p929zybtykyg/cage1-1_g0_t0.imec0.ap.bin?rlkey=ddltycvnjwzfragclzo0ad6yc&st=xa4vyagp&dl=0

https://drive.google.com/file/d/1-xMkbUBKOWLk5l3LpRjKn2su-N7Qv-YR/view?usp=drive_link

jacobpennington commented 1 month ago

Got it, thanks. I couldn't open the google drive link, I sent an access request for it from jacob.p.neuro@gmail.com. I'm not sure why my e-mail address wasn't showing up in the previous comment.