spikes found with KS2.0 but not with KS4

[JesusJBallesteros]

Describe the issue:

Hi there, thanks for all the work done!

The issue is the following. The exact same data files, sorted with KS2.0, seems to not be sorted with KS4.

The data is a couple bin files (int16) from different recording sessions, same probe map. I know they contain spikes, and they are easy to sort. Indeed KS2.0 find them without a problem, using our given settings.

My installation of KS4 seems to be working, as it is able to properly sort the example dataset as you provide it. Both, API and GUI versions.

However, when tried to sort our data in KS 4, it does not detect any spikes. Right after plotting the drift graphs (which are empty) the log reads: `0 spikes extracted in 76.21s; total 158.24s

Traceback (most recent call last): File "C:\code\miniconda3\envs\kilosort\lib\site-packages\kilosort\gui\sorter.py", line 82, in run st, tF, Wall0, clu0 = detect_spikes(ops, self.device, bfile, tic0=tic0, File "C:\code\miniconda3\envs\kilosort\lib\site-packages\kilosort\run_kilosort.py", line 397, in detect_spikes raise ValueError('No spikes detected, cannot continue sorting.')

ValueError : No spikes detected, cannot continue sorting..`

I tried several combinations of settings in KS4, from your defaults to sets alike to the one we use on KS2.0.

I am using the the GUI to interact with KS4, for now.

Here is an example of settings dump from KS4 in a latest test: {'data_file_path': WindowsPath('.../20240313/20240313.bin'), 'results_dir': WindowsPath('.../20240313/kilosort4'), 'probe': {'xc': array([ 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250., 50., 250.], dtype=float32), 'yc': array([ 50., 50., 100., 100., 150., 150., 200., 200., 250., 250., 300., 300., 350., 350., 400., 400., 450., 450., 500., 500., 550., 550., 600., 600., 650., 650., 700., 700., 750., 750., 800., 800.], dtype=float32), 'kcoords': array([1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2., 1., 2.], dtype=float32), 'chanMap': array([ 6, 5, 4, 3, 2, 1, 0, 31, 30, 29, 28, 27, 26, 25, 24, 23, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]), 'n_chan': 32}, 'probe_name': 'E32-S2.mat', 'data_dtype': 'int16', 'n_chan_bin': 32, 'fs': 32000.0, 'batch_size': 64000, 'nblocks': 1, 'Th_universal': 10.0, 'Th_learned': 4.0, 'tmin': 0.0, 'tmax': inf, 'nt': 31, 'artifact_threshold': inf, 'nskip': 1, 'whitening_range': 32, 'binning_depth': 5.0, 'sig_interp': 20.0, 'nt0min': None, 'dmin': None, 'dminx': None, 'min_template_size': 10.0, 'template_sizes': 5, 'nearest_chans': 10, 'nearest_templates': 100, 'templates_from_data': True, 'n_templates': 6, 'n_pcs': 6, 'Th_single_ch': 4.5, 'acg_threshold': 0.2, 'ccg_threshold': 0.25, 'cluster_downsampling': 20, 'cluster_pcs': 64, 'duplicate_spike_bins': 15}

Please, do you have any idea about what I might be missing? Thanks!

Edit: I've tried 'nt' as 61 and 31, but also as 65 as I though our 'fs' of 32KHz could be a reason to not match templates, with no differences on the output.

[jacobpennington]

Hello,

A couple things: first, drift correction is not likely to work well with that probe, so you could just set nblocks=0 to skip that step. It works best with, at minimum, ~32 channels on one shank (preferably more, and with denser spacing). @marius10p may be able to give more precise recommendations for that.

Second: several people have reported issues with multi-shank probes, which stem from the fact that we are not currently processing shanks separately and there are large gaps between them with no contacts. We're going to add support for this in the near future, but in the meantime you can try artificially stacking the two shanks vertically in the probe file with a bit of space between. I.e. change the values to: 'xc': array([50, 50, 50, ....]) and 'yc': array([50, 50+770, 100, 100+770, ...]) (770 being (max 800 - min 50) + a 20 micron gap)

I would suggest trying that first, then let us know if you still encounter issues.

[marius10p]

The channel spacing overall is a little high, and will indeed be suboptimal for drift correction, and also for isolating units from each other, but that's not the problem here. What probably does not work well is the automatic determination of channel groups.

@jacobpennington I think our strategy of making channel groups based on dmin and dminx needs to be explained somewhere and these parameters added to the GUI with tooltips. We should consider other strategies of making channel groups too, like changing it to "x nearest" channels, instead of channels within dmin and dminx distance, although I don't see a straightforward way right now to do that.

[jacobpennington]

@marius10p Which parameters? dmin and dminx are already in the GUI, with tooltips.

[marius10p]

Ok great, I guess I have to pull the latest version. I think we need a short guide somewhere about how to choose dmin, dminx for an arbitrary probe.

[JesusJBallesteros]

Thank you both for the suggestions.

in the meantime you can try artificially stacking the two shanks vertically in the probe file with a bit of space between.

This definitely did the trick. I'm looking forward to KS4 supporting low channel counts and far shanks, but this went relatively well.

drift correction is not likely to work well with that probe, so you could just set nblocks=0 to skip that step

This did not influence much on the output results. It worked when skipping drift calculation as well as with a non-zero nblock value. I would have to look into the small differences in spike count and clusters to see what's is more credible.

Great new version :)

[jacobpennington]

Glad that worked for now! Let us know if you run into any more problems with this, I'll close this for now.

As for skipping drift correction: there might be cases where the difference isn't noticeable, but in general doing drift correction with an insufficiently large & dense probe is likely to make things worse. For example, it may result in an estimate with large drift amounts that don't exist in the data.