CUDA invalid configuration argument

[bjhardcastle]

Run With Parameters 2082436

output:

                   ...
                   '1261.84 sec, 3101 batches, 23974785 spikes',
                   '1302.41 sec, 3201 batches, 24869319 spikes',
                   '1332.65 sec, 3275 batches, 25502528 spikes',
                   'time 2193.56, Shifted up/down 3275 batches.',
                   'Time 2223s. Optimizing templates ...',
                   '----------------------------------------Error using '
                   'gpuArray/nan',
                   'An unexpected error occurred trying to launch a kernel. '
                   'The CUDA error was:',
                   'invalid configuration argument'],
 'sorter_name': 'kilosort2_5',
 'sorter_version': 'compiled'}
SPIKE SORTING time: 4069.93s
...

[alejoe91]

In general KS CUDA errors are super hard to debug.. The only approach would be to try out different parameters and see if anything works. Since for that session it only happened for 1/6 probes, I would just ignore it and assume that this could happen every now and then :)

For this specific sessio, looking at the raster map, maybe this is due to the large artifact at ~1000s

We could take a closet look and try to find strategies to ecslude such events. @bjhardcastle what do you think?

[bjhardcastle]

That's really helpful, thanks. I didn't realise you could see the drift maps without the postprocessed output.

Raw data that corresponds to that artifact:

Is the a method in SI to specify intervals that should be ignored?

Along similar lines, we've been wondering about how we could exclude times when the opto laser is on to avoid artifacts being detected as spikes..

[alejoe91]

Oh you can see them for each session in the generated links: https://figurl.org/f?v=npm://@fi-sci/figurl-sortingview@12/dist&d=sha1://a558c3425650e4ef703771ce62ab1ee4c66e0a67&label=ecephys_690706_2023-11-28_12-25-50%20-%20experiment1_Record%20Node%20103%23Neuropix-PXI-100.ProbeA-AP_recording1&zone=aind

(see Drift Map tab)

If you record the signal driving the optical stimulation as an analog signal, I made a custom opto preprocessing capsule and pipeline that can be run.

In addition to the normal preprocessing, this adds a notch filter (because we observed a nasty 10k noise, probably from the external laser setup) and a remove artifacts step, where 2ms before and after each rising/falling edge are blanked out

[alejoe91]

The remove_artifacts function can remove any interval, but we need to find a proper way to pass its parameters...

[alejoe91]

@bjhardcastle I'm going through some older issues!

We now have a new Ecephys Trigger Capsule that allows you to select several parameters for the ephys pipeline(s).

Among these, you can specify t start and t stop to slice a portion of the recording. This could help targeted re-runs for cases like the one in this issue!