BUG: <KS4 does not detect spikes when sorting recordings with one long line of channels on sequential banks>

EricKenjiLee commented 3 months ago

Describe the issue:

I see this problem was previously brought up here but I'm running into errors when I also am trying to spike sort in KS4 using a "single long column". I'm using https://github.com/jenniferColonell/SGLXMetaToCoords/blob/main/SGLXMetaToCoords.py to convert the .imro to a KS4-readable .mat file and changing dmin = 20 and dminx = 103 to match probe geometry of NHP Longs. However, no spikes are detected in the recordings and processing fails. I suspect there is a setting here that I'm forgetting to change? Settings dump log attached.

Reproduce the bug:

This occurs when setting the dmin and dminx to 20 and 103 respectively on recordings acquired using a "single long channel bank" format for NHP long probes.

Error message:

Traceback (most recent call last):

  File "/home/chandlab/anaconda3/envs/kilosort4/lib/python3.9/site-packages/kilosort/gui/sorter.py", line 82, in run

st, tF, Wall0, clu0 = detect_spikes(ops, self.device, bfile, tic0=tic0,

  File "/home/chandlab/anaconda3/envs/kilosort4/lib/python3.9/site-packages/kilosort/run_kilosort.py", line 397, in detect_spikes

raise ValueError('No spikes detected, cannot continue sorting.')

ValueError
: 
No spikes detected, cannot continue sorting.

Version information:

Python: 3.9 Kilosort: 4.0.16 OS: Linux 20.04 CUDA toolkit: 11.5

EricKenjiLee commented 3 months ago

Adding settings dump settting_dump.log

jacobpennington commented 3 months ago

@EricKenjiLee Please try changing kcoords for the probe to have different values for the two columns of contacts. For example, 1 for contacts with xc == 11 and 2 for xc == 114. Treating those as a single shank will result in a lot of extra templates in the areas with no contacts, which could be causing the problem.

If you still encounter an error after trying that, please upload kilosort4.log from the results directory.

EricKenjiLee commented 3 months ago

Hey Jacob,

Thanks again for the help! I wrote in the change to kcoords (1's and 2's coordinating with the row indices of xc == 1 and xc == 2 respectively) but unfortunately it's still not detecting any drift or spikes. Attached is the log file.

Thanks again! Kenji

kilosort4.log

jacobpennington commented 3 months ago

@EricKenjiLee Sorry, I think I may not have worded my first reply very well. The values in probe['kc'] are still all 1's, but they should be different based on the value of probe['xc']. For example, if probe['xc'] starts with 11., 114., 11., 114., 11., 114., ... then probe['kc'] should start with 1, 2, 1, 2, 1, 2, ....

EricKenjiLee commented 3 months ago

No I think I understood you! These are the xcoords and kcoords variables in the .mat I am using. I think this is what you described.

jacobpennington commented 3 months ago

Yes that looks right, but the log file you uploaded is not using those values. Maybe it was the wrong log?

EricKenjiLee commented 3 months ago

Oops! Sorry here's the correct log file. It's showing the correct kcoords and still yielding the same error of not finding spikes. kilosort4.log

jacobpennington commented 3 months ago

Ah, sorry I didn't notice this in your first image: can you please try setting nblocks = 0 to turn off drift correction? I'm not sure why it would not be working with that probe layout, but it's estimating ~7000 microns of drift across all batches which is... strange.

Another thing to try, after trying it without drift correction, would be decreasing Th (universal) and Th (learned). I can see a few spikes in the screenshot of the data, but it looks like they're not standing out well from the background noise. Do you have a sense of whether the recording itself is exceptionally noisy?

EricKenjiLee commented 3 months ago

Hmm I just tried both of those things and the recordings still fail to find any spikes! I don't find that these recordings are any noisier than usual and when I spike sort each bank individually (by disabling all the channels of a different bank), I find that KS4 works just fine and finds lots of good units. I'm not sure what else to try; the drift estimate on each probe individually is around 20 microns so nothing out of the ordinary.

jacobpennington commented 1 month ago

@EricKenjiLee Sorry for the delayed response, things have been very busy lately. Would you be willing to share a sample dataset that you see this problem with? I have an idea of what might be causing it, but I can't confirm it without some data that reproduces the issue.

EricKenjiLee commented 1 month ago

No worries! We just kilosort each contiguous stretch of channels separately. Obviously not ideal but also doesn't seem to be a huge issue other than possible double counting of units. I'm attaching a 10 minute recording that hasn't been sorted but uses the channel mapping as described.

https://www.dropbox.com/scl/fi/6t1gvjfnuc5pue95ud21q/TIBERIUS_AUDOVSL_DLPFC_20240821_g0.zip?rlkey=ufp94h0fuha25u5m45m26nvbq&st=gc8y6h76&dl=0

jacobpennington commented 2 weeks ago

Sorry for the delayed response. I've confirmed the problem is stemming from the default template placement behavior. In short, setting dminx = 103 (as you should) results in a column of template positions right down the middle of the probe where there are no channels, which causes spike detection to fail. I'm working on a fix for that, but the most straightforward solution in the meantime is to set max_channel_distance = 20 to remove those extra templates. Using the default dminx = 32 also works, but may run a bit slower.

EricKenjiLee commented 1 week ago

Hey Jacob,

No worries at all! Happy to report that your fix worked perfectly as far as we can tell. Feel free to close the issue if you want!

Thanks so much! Kenji

MouseLand / Kilosort