ProbeInterface or Spikeinterface issue?

[aghatpande]

I'm using SpikeInterface version 0.100.5 with Probeinterface version 0.2.21. I have recorded data with Cambridge Neurotech E2 series probe, which I imported into Probeinterface. This executes successfully and I can print the probe map etc. When I attach the probe to the recording it doesn't work:

# set the probe to recording_saved
recording_saved.set_probe(cnE2probe)

ZarrRecordingExtractor: 64 channels - 32.0kHz - 1 segments - 19,036,223 samples 594.88s (9.91 minutes) - int16 dtype - 2.27 GiB

# did it work?
# check if the probe is attached
print (f'Does the recording have a probe attached? : {recording_saved.has_probe()}')

Does the recording have a probe attached? : False

Apparently I need to attach a probe so as to sort the data using Kilosort4 (via Docker) although when I did try KS4 it only issued a warning, created a dummy probe based on channel locations, apparently sorted the data 100% (progress bar) and then ultimately failed! I have successfully sorted this same data with KS3 using SI and Docker. Any advice is most appreciated.

[alejoe91]

The set_probe returns a new object with the attached probe. This is because the probe could contain a subset of channels, so the new recording will have a fewer number of channels.

So you should do:

recording_w_probe = recording_saved.set_probe(cnE2probe)
print (f'Does the recording have a probe attached? : {recording_w_probe.has_probe()}')

[aghatpande]

Thanks Alessio, the probe attaches to the new recording object, so that helped me. But Kilosort4 still complains of not finding the probe,seems like tries to go through the entire data and then quits. Apologies for the extremely long error message, feel free to delete it. Your advice most appreciated. See below:

# run KS4 on entire recording from docker container
sorting_KS4 = si.run_sorter('kilosort4', recording_w, 
                            output_folder=base_folder / 'results__nopre_rec1_2_KS4_0drift',
                            verbose=True,
                            docker_image=True)`

{
    "name": "SpikeSortingError",
    "message": "Spike sorting in docker failed with the following error:
/root/.local/lib/python3.11/site-packages/spikeinterface/core/baserecordingsnippets.py:244: UserWarning: There is no Probe attached to this recording. Creating a dummy one with contact positions
  warn(\"There is no Probe attached to this recording. Creating a dummy one with contact positions\")
========================================
Loading recording with SpikeInterface...
number of samples: 19036223
number of channels: 64
numbef of segments: 1
sampling rate: 32000.0
dtype: int16
========================================
Preprocessing filters computed in  2.85s; total  2.85s

computing drift
Re-computing universal templates from data.
\r  0%|          | 0/318 [00:00<?, ?it/s]\r  0%|          | 1/318 [00:00<01:09,  4.57it/s]\r  1%|          | 2/318 [00:00<01:06,  4.73it/s]\r  1%|          | 3/318 [00:00<01:05,  4.84it/s]\r  1%|▏         | 4/318 [00:00<01:08,  4.60it/s]\r  2%|▏         | 5/318 [00:01<01:10,  4.45it/s]\r  2%|▏         | 6/318 [00:01<01:08,  4.54it/s]\r  2%|▏         | 7/318 [00:01<01:06,  4.67it/s]\r  3%|▎         | 8/318 [00:01<01:04,  4.82it/s]\r  3%|▎         | 9/318 [00:01<01:07,  4.59it/s]\r  3%|▎         | 10/318 [00:02<01:08,  4.50it/s]\r  3%|▎         | 11/318 [00:02<01:06,  4.64it/s]\r  4%|▍         | 12/318 [00:02<01:04,  4.73it/s]\r  4%|▍         | 13/318 [00:02<01:03,  4.81it/s]\r  4%|▍         | 14/318 [00:02<01:01,  4.91it/s]\r  5%|▍         | 15/318 [00:03<00:59,  5.12it/s]\r  5%|▌         | 16/318 [00:03<00:57,  5.23it/s]\r  5%|▌         | 17/318 [00:03<00:57,  5.23it/s]\r  6%|▌         | 18/318 [00:03<00:57,  5.25it/s]\r  6%|▌         | 19/318 [00:03<00:57,  5.23it/s]\r  6%|▋         | 20/318 [00:04<00:56,  5.24it/s]\r  7%|▋         | 21/318 [00:04<00:58,  5.04it/s]\r  7%|▋         | 22/318 [00:04<00:56,  5.24it/s]\r  7%|▋         | 23/318 [00:04<00:53,  5.51it/s]\r  8%|▊         | 24/318 [00:04<00:53,  5.47it/s]\r  8%|▊         | 25/318 [00:05<00:54,  5.33it/s]\r  8%|▊         | 26/318 [00:05<00:55,  5.22it/s]\r  8%|▊         | 27/318 [00:05<00:56,  5.12it/s]\r  9%|▉         | 28/318 [00:05<00:57,  5.04it/s]\r  9%|▉         | 29/318 [00:05<01:01,  4.74it/s]\r  9%|▉         | 30/318 [00:06<01:03,  4.56it/s]\r 10%|▉         | 31/318 [00:06<01:01,  4.66it/s]\r 10%|█         | 32/318 [00:06<01:02,  4.57it/s]\r 10%|█         | 33/318 [00:06<01:01,  4.67it/s]\r 11%|█         | 34/318 [00:06<01:00,  4.67it/s]\r 11%|█         | 35/318 [00:07<01:00,  4.67it/s]\r 11%|█▏        | 36/318 [00:07<00:59,  4.74it/s]\r 12%|█▏        | 37/318 [00:07<00:58,  4.80it/s]\r 12%|█▏        | 38/318 [00:07<00:58,  4.79it/s]\r 12%|█▏        | 39/318 [00:08<00:57,  4.81it/s]\r 13%|█▎        | 40/318 [00:08<01:00,  4.63it/s]\r 13%|█▎        | 41/318 [00:08<00:58,  4.70it/s]\r 13%|█▎        | 42/318 [00:08<00:59,  4.68it/s]\r 14%|█▎        | 43/318 [00:08<00:56,  4.82it/s]\r 14%|█▍        | 44/318 [00:09<00:57,  4.76it/s]\r 14%|█▍        | 45/318 [00:09<00:58,  4.70it/s]\r 14%|█▍        | 46/318 [00:09<00:53,  5.09it/s]\r 15%|█▍        | 47/318 [00:09<00:53,  5.08it/s]\r 15%|█▌        | 48/318 [00:09<00:51,  5.23it/s]\r 15%|█▌        | 49/318 [00:10<00:53,  5.01it/s]\r 16%|█▌        | 50/318 [00:10<00:56,  4.72it/s]\r 16%|█▌        | 51/318 [00:10<00:56,  4.75it/s]\r 16%|█▋        | 52/318 [00:10<00:56,  4.73it/s]\r 17%|█▋        | 53/318 [00:10<00:56,  4.66it/s]\r 17%|█▋        | 54/318 [00:11<00:56,  4.65it/s]\r 17%|█▋        | 55/318 [00:11<00:56,  4.68it/s]\r 18%|█▊        | 56/318 [00:11<00:55,  4.71it/s]\r 18%|█▊        | 57/318 [00:11<00:53,  4.91it/s]\r 18%|█▊        | 58/318 [00:11<00:52,  4.93it/s]\r 19%|█▊        | 59/318 [00:12<00:51,  5.02it/s]\r 19%|█▉        | 60/318 [00:12<00:51,  5.02it/s]\r 19%|█▉        | 61/318 [00:12<00:53,  4.77it/s]\r 19%|█▉        | 62/318 [00:12<00:50,  5.03it/s]\r 20%|█▉        | 63/318 [00:12<00:48,  5.24it/s]\r 20%|██        | 64/318 [00:13<00:49,  5.11it/s]\r 20%|██        | 65/318 [00:13<00:50,  5.05it/s]\r 21%|██        | 66/318 [00:13<00:51,  4.94it/s]\r 21%|██        | 67/318 [00:13<00:51,  4.90it/s]\r 21%|██▏       | 68/318 [00:13<00:49,  5.02it/s]\r 22%|██▏       | 69/318 [00:14<00:49,  5.05it/s]\r 22%|██▏       | 70/318 [00:14<00:52,  4.77it/s]\r 22%|██▏       | 71/318 [00:14<00:50,  4.85it/s]\r 23%|██▎       | 72/318 [00:14<00:44,  5.49it/s]\r 23%|██▎       | 73/318 [00:14<00:45,  5.34it/s]\r 23%|██▎       | 74/318 [00:15<00:47,  5.17it/s]\r 24%|██▎       | 75/318 [00:15<00:47,  5.08it/s]\r 24%|██▍       | 76/318 [00:15<00:48,  4.99it/s]\r 24%|██▍       | 77/318 [00:15<00:48,  4.96it/s]\r 25%|██▍       | 78/318 [00:15<00:49,  4.88it/s]\r 25%|██▍       | 79/318 [00:16<00:48,  4.98it/s]\r 25%|██▌       | 80/318 [00:16<00:49,  4.82it/s]\r 25%|██▌       | 81/318 [00:16<00:49,  4.79it/s]\r 26%|██▌       | 82/318 [00:16<00:51,  4.59it/s]\r 26%|██▌       | 83/318 [00:16<00:48,  4.84it/s]\r 26%|██▋       | 84/318 [00:17<00:47,  4.93it/s]\r 27%|██▋       | 85/318 [00:17<00:48,  4.78it/s]\r 27%|██▋       | 86/318 [00:17<00:47,  4.86it/s]\r 27%|██▋       | 87/318 [00:17<00:48,  4.76it/s]\r 28%|██▊       | 88/318 [00:18<00:47,  4.83it/s]\r 28%|██▊       | 89/318 [00:18<00:46,  4.88it/s]\r 28%|██▊       | 90/318 [00:18<00:46,  4.94it/s]\r 29%|██▊       | 91/318 [00:18<00:47,  4.74it/s]\r 29%|██▉       | 92/318 [00:18<00:46,  4.82it/s]\r 29%|██▉       | 93/318 [00:19<00:47,  4.70it/s]\r 30%|██▉       | 94/318 [00:19<00:46,  4.81it/s]\r 30%|██▉       | 95/318 [00:19<00:45,  4.85it/s]\r 30%|███       | 96/318 [00:19<00:45,  4.90it/s]\r 31%|███       | 97/318 [00:19<00:44,  4.94it/s]\r 31%|███       | 98/318 [00:20<00:46,  4.69it/s]\r 31%|███       | 99/318 [00:20<00:47,  4.64it/s]\r 31%|███▏      | 100/318 [00:20<00:48,  4.52it/s]\r 32%|███▏      | 101/318 [00:20<00:49,  4.43it/s]\r 32%|███▏      | 102/318 [00:20<00:47,  4.53it/s]\r 32%|███▏      | 103/318 [00:21<00:46,  4.61it/s]\r 33%|███▎      | 104/318 [00:21<00:46,  4.65it/s]\r 33%|███▎      | 105/318 [00:21<00:46,  4.54it/s]\r 33%|███▎      | 106/318 [00:21<00:47,  4.44it/s]\r 34%|███▎      | 107/318 [00:22<00:45,  4.61it/s]\r 34%|███▍      | 108/318 [00:22<00:44,  4.68it/s]\r 34%|███▍      | 109/318 [00:22<00:45,  4.56it/s]\r 35%|███▍      | 110/318 [00:22<00:44,  4.66it/s]\r 35%|███▍      | 111/318 [00:22<00:44,  4.70it/s]\r 35%|███▌      | 112/318 [00:23<00:42,  4.90it/s]\r 36%|███▌      | 113/318 [00:23<00:42,  4.82it/s]\r 36%|███▌      | 114/318 [00:23<00:41,  4.88it/s]\r 36%|███▌      | 115/318 [00:23<00:43,  4.69it/s]\r 36%|███▋      | 116/318 [00:23<00:44,  4.55it/s]\r 37%|███▋      | 117/318 [00:24<00:43,  4.65it/s]\r 37%|███▋      | 118/318 [00:24<00:41,  4.79it/s]\r 37%|███▋      | 119/318 [00:24<00:40,  4.88it/s]\r 38%|███▊      | 120/318 [00:24<00:39,  4.95it/s]\r 38%|███▊      | 121/318 [00:24<00:39,  5.05it/s]\r 38%|███▊      | 122/318 [00:25<00:39,  4.98it/s]\r 39%|███▊      | 123/318 [00:25<00:39,  4.94it/s]\r 39%|███▉      | 124/318 [00:25<00:39,  4.92it/s]\r 39%|███▉      | 125/318 [00:25<00:41,  4.67it/s]\r 40%|███▉      | 126/318 [00:26<00:39,  4.86it/s]\r 40%|███▉      | 127/318 [00:26<00:38,  4.93it/s]\r 40%|████      | 128/318 [00:26<00:40,  4.70it/s]\r 41%|████      | 129/318 [00:26<00:40,  4.69it/s]\r 41%|████      | 130/318 [00:26<00:39,  4.74it/s]\r 41%|████      | 131/318 [00:27<00:36,  5.13it/s]\r 42%|████▏     | 132/318 [00:27<00:36,  5.15it/s]\r 42%|████▏     | 133/318 [00:27<00:36,  5.07it/s]\r 42%|████▏     | 134/318 [00:27<00:36,  5.08it/s]\r 42%|████▏     | 135/318 [00:27<00:38,  4.81it/s]\r 43%|████▎     | 136/318 [00:28<00:37,  4.85it/s]\r 43%|████▎     | 137/318 [00:28<00:37,  4.84it/s]\r 43%|████▎     | 138/318 [00:28<00:38,  4.63it/s]\r 44%|████▎     | 139/318 [00:28<00:37,  4.72it/s]\r 44%|████▍     | 140/318 [00:28<00:37,  4.73it/s]\r 44%|████▍     | 141/318 [00:29<00:37,  4.76it/s]\r 45%|████▍     | 142/318 [00:29<00:37,  4.74it/s]\r 45%|████▍     | 143/318 [00:29<00:35,  4.95it/s]\r 45%|████▌     | 144/318 [00:29<00:34,  5.00it/s]\r 46%|████▌     | 145/318 [00:29<00:34,  4.97it/s]\r 46%|████▌     | 146/318 [00:30<00:36,  4.70it/s]\r 46%|████▌     | 147/318 [00:30<00:34,  4.91it/s]\r 47%|████▋     | 148/318 [00:30<00:34,  4.89it/s]\r 47%|████▋     | 149/318 [00:30<00:35,  4.70it/s]\r 47%|████▋     | 150/318 [00:30<00:35,  4.69it/s]\r 47%|████▋     | 151/318 [00:31<00:35,  4.67it/s]\r 48%|████▊     | 152/318 [00:31<00:35,  4.70it/s]\r 48%|████▊     | 153/318 [00:31<00:35,  4.70it/s]\r 48%|████▊     | 154/318 [00:31<00:34,  4.70it/s]\r 49%|████▊     | 155/318 [00:32<00:34,  4.79it/s]\r 49%|████▉     | 156/318 [00:32<00:34,  4.64it/s]\r 49%|████▉     | 157/318 [00:32<00:36,  4.45it/s]\r 50%|████▉     | 158/318 [00:32<00:35,  4.54it/s]\r 50%|█████     | 159/318 [00:32<00:34,  4.61it/s]\r 50%|█████     | 160/318 [00:33<00:34,  4.62it/s]\r 51%|█████     | 161/318 [00:33<00:33,  4.65it/s]\r 51%|█████     | 162/318 [00:33<00:35,  4.44it/s]\r 51%|█████▏    | 163/318 [00:33<00:34,  4.48it/s]\r 52%|█████▏    | 164/318 [00:34<00:35,  4.34it/s]\r 52%|█████▏    | 165/318 [00:34<00:34,  4.46it/s]\r 52%|█████▏    | 166/318 [00:34<00:32,  4.62it/s]\r 53%|█████▎    | 167/318 [00:34<00:30,  4.89it/s]\r 53%|█████▎    | 168/318 [00:34<00:29,  5.04it/s]\r 53%|█████▎    | 169/318 [00:35<00:30,  4.87it/s]\r 53%|█████▎    | 170/318 [00:35<00:30,  4.85it/s]\r 54%|█████▍    | 171/318 [00:35<00:30,  4.78it/s]\r 54%|█████▍    | 172/318 [00:35<00:30,  4.79it/s]\r 54%|█████▍    | 173/318 [00:35<00:29,  4.87it/s]\r 55%|█████▍    | 174/318 [00:36<00:30,  4.68it/s]\r 55%|█████▌    | 175/318 [00:36<00:30,  4.72it/s]\r 55%|█████▌    | 176/318 [00:36<00:30,  4.71it/s]\r 56%|█████▌    | 177/318 [00:36<00:30,  4.67it/s]\r 56%|█████▌    | 178/318 [00:36<00:30,  4.66it/s]\r 56%|█████▋    | 179/318 [00:37<00:29,  4.66it/s]\r 57%|█████▋    | 180/318 [00:37<00:30,  4.55it/s]\r 57%|█████▋    | 181/318 [00:37<00:30,  4.45it/s]\r 57%|█████▋    | 182/318 [00:37<00:29,  4.54it/s]\r 58%|█████▊    | 183/318 [00:38<00:30,  4.43it/s]\r 58%|█████▊    | 184/318 [00:38<00:30,  4.39it/s]\r 58%|█████▊    | 185/318 [00:38<00:29,  4.53it/s]\r 58%|█████▊    | 186/318 [00:38<00:28,  4.65it/s]\r 59%|█████▉    | 187/318 [00:38<00:28,  4.66it/s]\r 59%|█████▉    | 188/318 [00:39<00:27,  4.76it/s]\r 59%|█████▉    | 189/318 [00:39<00:26,  4.88it/s]\r 60%|█████▉    | 190/318 [00:39<00:27,  4.73it/s]\r 60%|██████    | 191/318 [00:39<00:26,  4.84it/s]\r 60%|██████    | 192/318 [00:39<00:25,  4.99it/s]\r 61%|██████    | 193/318 [00:40<00:25,  4.96it/s]\r 61%|██████    | 194/318 [00:40<00:24,  4.98it/s]\r 61%|██████▏   | 195/318 [00:40<00:24,  5.02it/s]\r 62%|██████▏   | 196/318 [00:40<00:24,  4.99it/s]\r 62%|██████▏   | 197/318 [00:40<00:23,  5.09it/s]\r 62%|██████▏   | 198/318 [00:41<00:24,  4.81it/s]\r 63%|██████▎   | 199/318 [00:41<00:24,  4.79it/s]\r 63%|██████▎   | 200/318 [00:41<00:24,  4.88it/s]\r 63%|██████▎   | 201/318 [00:41<00:24,  4.83it/s]\r 64%|██████▎   | 202/318 [00:42<00:23,  4.87it/s]\r 64%|██████▍   | 203/318 [00:42<00:24,  4.65it/s]\r 64%|██████▍   | 204/318 [00:42<00:24,  4.65it/s]\r 64%|██████▍   | 205/318 [00:42<00:24,  4.63it/s]\r 65%|██████▍   | 206/318 [00:42<00:24,  4.49it/s]\r 65%|██████▌   | 207/318 [00:43<00:24,  4.56it/s]\r 65%|██████▌   | 208/318 [00:43<00:23,  4.64it/s]\r 66%|██████▌   | 209/318 [00:43<00:23,  4.67it/s]\r 66%|██████▌   | 210/318 [00:43<00:23,  4.67it/s]\r 66%|██████▋   | 211/318 [00:43<00:22,  4.70it/s]\r 67%|██████▋   | 212/318 [00:44<00:23,  4.52it/s]\r 67%|██████▋   | 213/318 [00:44<00:22,  4.63it/s]\r 67%|██████▋   | 214/318 [00:44<00:21,  4.78it/s]\r 68%|██████▊   | 215/318 [00:44<00:22,  4.65it/s]\r 68%|██████▊   | 216/318 [00:45<00:21,  4.83it/s]\r 68%|██████▊   | 217/318 [00:45<00:20,  4.96it/s]\r 69%|██████▊   | 218/318 [00:45<00:20,  4.90it/s]\r 69%|██████▉   | 219/318 [00:45<00:20,  4.90it/s]\r 69%|██████▉   | 220/318 [00:45<00:19,  4.96it/s]\r 69%|██████▉   | 221/318 [00:46<00:19,  5.03it/s]\r 70%|██████▉   | 222/318 [00:46<00:18,  5.05it/s]\r 70%|███████   | 223/318 [00:46<00:18,  5.03it/s]\r 70%|███████   | 224/318 [00:46<00:18,  5.06it/s]\r 71%|███████   | 225/318 [00:46<00:19,  4.86it/s]\r 71%|███████   | 226/318 [00:47<00:18,  4.93it/s]\r 71%|███████▏  | 227/318 [00:47<00:18,  4.91it/s]\r 72%|███████▏  | 228/318 [00:47<00:18,  4.96it/s]\r 72%|███████▏  | 229/318 [00:47<00:18,  4.90it/s]\r 72%|███████▏  | 230/318 [00:47<00:18,  4.77it/s]\r 73%|███████▎  | 231/318 [00:48<00:18,  4.79it/s]\r 73%|███████▎  | 232/318 [00:48<00:17,  4.93it/s]\r 73%|███████▎  | 233/318 [00:48<00:17,  4.93it/s]\r 74%|███████▎  | 234/318 [00:48<00:17,  4.81it/s]\r 74%|███████▍  | 235/318 [00:48<00:17,  4.85it/s]\r 74%|███████▍  | 236/318 [00:49<00:17,  4.68it/s]\r 75%|███████▍  | 237/318 [00:49<00:16,  4.80it/s]\r 75%|███████▍  | 238/318 [00:49<00:16,  4.89it/s]\r 75%|███████▌  | 239/318 [00:49<00:16,  4.88it/s]\r 75%|███████▌  | 240/318 [00:49<00:16,  4.64it/s]\r 76%|███████▌  | 241/318 [00:50<00:16,  4.75it/s]\r 76%|███████▌  | 242/318 [00:50<00:15,  4.89it/s]\r 76%|███████▋  | 243/318 [00:50<00:15,  4.76it/s]\r 77%|███████▋  | 244/318 [00:50<00:15,  4.74it/s]\r 77%|███████▋  | 245/318 [00:50<00:15,  4.81it/s]\r 77%|███████▋  | 246/318 [00:51<00:14,  4.84it/s]\r 78%|███████▊  | 247/318 [00:51<00:14,  4.84it/s]\r 78%|███████▊  | 248/318 [00:51<00:14,  4.85it/s]\r 78%|███████▊  | 249/318 [00:51<00:14,  4.84it/s]\r 79%|███████▊  | 250/318 [00:52<00:14,  4.70it/s]\r 79%|███████▉  | 251/318 [00:52<00:14,  4.63it/s]\r 79%|███████▉  | 252/318 [00:52<00:14,  4.71it/s]\r 80%|███████▉  | 253/318 [00:52<00:13,  4.75it/s]\r 80%|███████▉  | 254/318 [00:52<00:13,  4.76it/s]\r 80%|████████  | 255/318 [00:53<00:13,  4.59it/s]\r 81%|████████  | 256/318 [00:53<00:13,  4.70it/s]\r 81%|████████  | 257/318 [00:53<00:12,  4.93it/s]\r 81%|████████  | 258/318 [00:53<00:11,  5.07it/s]\r 81%|████████▏ | 259/318 [00:53<00:11,  5.18it/s]\r 82%|████████▏ | 260/318 [00:54<00:11,  5.10it/s]\r 82%|████████▏ | 261/318 [00:54<00:11,  5.03it/s]\r 82%|████████▏ | 262/318 [00:54<00:11,  5.09it/s]\r 83%|████████▎ | 263/318 [00:54<00:10,  5.15it/s]\r 83%|████████▎ | 264/318 [00:54<00:10,  5.07it/s]\r 83%|████████▎ | 265/318 [00:55<00:10,  5.07it/s]\r 84%|████████▎ | 266/318 [00:55<00:10,  4.99it/s]\r 84%|████████▍ | 267/318 [00:55<00:09,  5.14it/s]\r 84%|████████▍ | 268/318 [00:55<00:09,  5.22it/s]\r 85%|████████▍ | 269/318 [00:55<00:09,  5.25it/s]\r 85%|████████▍ | 270/318 [00:55<00:09,  5.32it/s]\r 85%|████████▌ | 271/318 [00:56<00:08,  5.33it/s]\r 86%|████████▌ | 272/318 [00:56<00:08,  5.17it/s]\r 86%|████████▌ | 273/318 [00:56<00:08,  5.22it/s]\r 86%|████████▌ | 274/318 [00:56<00:08,  5.06it/s]\r 86%|████████▋ | 275/318 [00:56<00:08,  5.09it/s]\r 87%|████████▋ | 276/318 [00:57<00:08,  5.09it/s]\r 87%|████████▋ | 277/318 [00:57<00:08,  5.01it/s]\r 87%|████████▋ | 278/318 [00:57<00:08,  4.95it/s]\r 88%|████████▊ | 279/318 [00:57<00:07,  4.95it/s]\r 88%|████████▊ | 280/318 [00:58<00:07,  4.94it/s]\r 88%|████████▊ | 281/318 [00:58<00:07,  4.95it/s]\r 89%|████████▊ | 282/318 [00:58<00:07,  5.00it/s]\r 89%|████████▉ | 283/318 [00:58<00:07,  4.86it/s]\r 89%|████████▉ | 284/318 [00:58<00:07,  4.80it/s]\r 90%|████████▉ | 285/318 [00:59<00:06,  4.76it/s]\r 90%|████████▉ | 286/318 [00:59<00:06,  4.64it/s]\r 90%|█████████ | 287/318 [00:59<00:06,  4.69it/s]\r 91%|█████████ | 288/318 [00:59<00:06,  4.87it/s]\r 91%|█████████ | 289/318 [00:59<00:06,  4.79it/s]\r 91%|█████████ | 290/318 [01:00<00:05,  4.87it/s]\r 92%|█████████▏| 291/318 [01:00<00:05,  4.96it/s]\r 92%|█████████▏| 292/318 [01:00<00:05,  4.91it/s]\r 92%|█████████▏| 293/318 [01:00<00:04,  5.05it/s]\r 92%|█████████▏| 294/318 [01:00<00:04,  5.12it/s]\r 93%|█████████▎| 295/318 [01:01<00:04,  5.10it/s]\r 93%|█████████▎| 296/318 [01:01<00:04,  5.09it/s]\r 93%|█████████▎| 297/318 [01:01<00:04,  5.07it/s]\r 94%|█████████▎| 298/318 [01:01<00:04,  4.99it/s]\r 94%|█████████▍| 299/318 [01:01<00:03,  4.96it/s]\r 94%|█████████▍| 300/318 [01:02<00:03,  4.87it/s]\r 95%|█████████▍| 301/318 [01:02<00:03,  4.93it/s]\r 95%|█████████▍| 302/318 [01:02<00:03,  4.96it/s]\r 95%|█████████▌| 303/318 [01:02<00:03,  4.83it/s]\r 96%|█████████▌| 304/318 [01:02<00:02,  4.75it/s]\r 96%|█████████▌| 305/318 [01:03<00:02,  4.88it/s]\r 96%|█████████▌| 306/318 [01:03<00:02,  4.98it/s]\r 97%|█████████▋| 307/318 [01:03<00:02,  5.14it/s]\r 97%|█████████▋| 308/318 [01:03<00:01,  5.29it/s]\r 97%|█████████▋| 309/318 [01:03<00:01,  5.34it/s]\r 97%|█████████▋| 310/318 [01:04<00:01,  5.35it/s]\r 98%|█████████▊| 311/318 [01:04<00:01,  5.19it/s]\r 98%|█████████▊| 312/318 [01:04<00:01,  5.19it/s]\r 98%|█████████▊| 313/318 [01:04<00:00,  5.25it/s]\r 99%|█████████▊| 314/318 [01:04<00:00,  5.23it/s]\r 99%|█████████▉| 315/318 [01:04<00:00,  5.32it/s]\r 99%|█████████▉| 316/318 [01:05<00:00,  5.38it/s]\r100%|█████████▉| 317/318 [01:05<00:00,  5.36it/s]\r100%|██████████| 318/318 [01:05<00:00,  6.03it/s]\r100%|██████████| 318/318 [01:05<00:00,  4.86it/s]
Traceback (most recent call last):
  File \"/Users/aghat/Documents/analysis/chenlab_organoids/Dec14_2023/in_container_sorter_script.py\", line 23, in <module>
drift computed in  68.67s; total  71.52s

Extracting spikes using templates
Re-computing universal templates from data.
Error running kilosort4
    sorting = run_sorter_local(
              ^^^^^^^^^^^^^^^^^
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py\", line 225, in run_sorter_local
    SorterClass.run_from_folder(output_folder, raise_error, verbose)
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py\", line 293, in run_from_folder
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py\", line 258, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/external/kilosort4.py\", line 260, in _run_from_folder
    st, tF, _, _ = detect_spikes(ops, device, bfile, tic0=tic0, progress_bar=progress_bar)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/kilosort/run_kilosort.py\", line 392, in detect_spikes
    st0, tF, ops = spikedetect.run(ops, bfile, device=device, progress_bar=progress_bar)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py\", line 193, in run
    ops['wPCA'], ops['wTEMP'] = extract_wPCA_wTEMP(
                                ^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py\", line 70, in extract_wPCA_wTEMP
    model = TruncatedSVD(n_components=ops['settings']['n_pcs']).fit(clips)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/decomposition/_truncated_svd.py\", line 209, in fit
    self.fit_transform(X)
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/utils/_set_output.py\", line 295, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/base.py\", line 1474, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/decomposition/_truncated_svd.py\", line 229, in fit_transform
    X = self._validate_data(X, accept_sparse=[\"csr\", \"csc\"], ensure_min_features=2)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/base.py\", line 633, in _validate_data
    out = check_array(X, input_name=\"X\", **check_params)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/utils/validation.py\", line 1072, in check_array
    raise ValueError(
ValueError: Found array with 0 sample(s) (shape=(0, 61)) while a minimum of 1 is required by TruncatedSVD.

Spike sorting failed. You can inspect the runtime trace in /Users/aghat/Documents/analysis/chenlab_organoids/Dec14_2023/results__nopre_rec1_2_KS4_0drift/spikeinterface_log.json.
",
    "stack": "---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Cell In[70], line 2
      1 # run KS4 on entire recording from docker container
----> 2 sorting_KS4 = si.run_sorter('kilosort4', recording_w, 
      3                             output_folder=base_folder / 'results__nopre_rec1_2_KS4_0drift',
      4                             verbose=True,
      5                             docker_image=True)

File c:\\Users\\aghat\\anaconda3\\envs\\si100dt4_pynapple\\lib\\site-packages\\spikeinterface\\sorters\\runsorter.py:169, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    167         else:
    168             container_image = singularity_image
--> 169     return run_sorter_container(
    170         container_image=container_image,
    171         mode=mode,
    172         **common_kwargs,
    173     )
    175 return run_sorter_local(**common_kwargs)

File c:\\Users\\aghat\\anaconda3\\envs\\si100dt4_pynapple\\lib\\site-packages\\spikeinterface\\sorters\\runsorter.py:595, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, delete_container_files, extra_requirements, installation_mode, spikeinterface_version, spikeinterface_folder_source, **sorter_params)
    593 if run_error:
    594     if raise_error:
--> 595         raise SpikeSortingError(f\"Spike sorting in {mode} failed with the following error:\
{run_sorter_output}\")
    596 else:
    597     if with_output:

SpikeSortingError: Spike sorting in docker failed with the following error:
/root/.local/lib/python3.11/site-packages/spikeinterface/core/baserecordingsnippets.py:244: UserWarning: There is no Probe attached to this recording. Creating a dummy one with contact positions
  warn(\"There is no Probe attached to this recording. Creating a dummy one with contact positions\")
========================================
Loading recording with SpikeInterface...
number of samples: 19036223
number of channels: 64
numbef of segments: 1
sampling rate: 32000.0
dtype: int16
========================================
Preprocessing filters computed in  2.85s; total  2.85s

computing drift
Re-computing universal templates from data.
\r  0%|          | 0/318 [00:00<?, ?it/s]\r  0%|          | 1/318 [00:00<01:09,  4.57it/s]\r  1%|          | 2/318 [00:00<01:06,  4.73it/s]\r  1%|          | 3/318 [00:00<01:05,  4.84it/s]\r  1%|▏         | 4/318 [00:00<01:08,  4.60it/s]\r  2%|▏         | 5/318 [00:01<01:10,  4.45it/s]\r  2%|▏         | 6/318 [00:01<01:08,  4.54it/s]\r  2%|▏         | 7/318 [00:01<01:06,  4.67it/s]\r  3%|▎         | 8/318 [00:01<01:04,  4.82it/s]\r  3%|▎         | 9/318 [00:01<01:07,  4.59it/s]\r  3%|▎         | 10/318 [00:02<01:08,  4.50it/s]\r  3%|▎         | 11/318 [00:02<01:06,  4.64it/s]\r  4%|▍         | 12/318 [00:02<01:04,  4.73it/s]\r  4%|▍         | 13/318 [00:02<01:03,  4.81it/s]\r  4%|▍         | 14/318 [00:02<01:01,  4.91it/s]\r  5%|▍         | 15/318 [00:03<00:59,  5.12it/s]\r  5%|▌         | 16/318 [00:03<00:57,  5.23it/s]\r  5%|▌         | 17/318 [00:03<00:57,  5.23it/s]\r  6%|▌         | 18/318 [00:03<00:57,  5.25it/s]\r  6%|▌         | 19/318 [00:03<00:57,  5.23it/s]\r  6%|▋         | 20/318 [00:04<00:56,  5.24it/s]\r  7%|▋         | 21/318 [00:04<00:58,  5.04it/s]\r  7%|▋         | 22/318 [00:04<00:56,  5.24it/s]\r  7%|▋         | 23/318 [00:04<00:53,  5.51it/s]\r  8%|▊         | 24/318 [00:04<00:53,  5.47it/s]\r  8%|▊         | 25/318 [00:05<00:54,  5.33it/s]\r  8%|▊         | 26/318 [00:05<00:55,  5.22it/s]\r  8%|▊         | 27/318 [00:05<00:56,  5.12it/s]\r  9%|▉         | 28/318 [00:05<00:57,  5.04it/s]\r  9%|▉         | 29/318 [00:05<01:01,  4.74it/s]\r  9%|▉         | 30/318 [00:06<01:03,  4.56it/s]\r 10%|▉         | 31/318 [00:06<01:01,  4.66it/s]\r 10%|█         | 32/318 [00:06<01:02,  4.57it/s]\r 10%|█         | 33/318 [00:06<01:01,  4.67it/s]\r 11%|█         | 34/318 [00:06<01:00,  4.67it/s]\r 11%|█         | 35/318 [00:07<01:00,  4.67it/s]\r 11%|█▏        | 36/318 [00:07<00:59,  4.74it/s]\r 12%|█▏        | 37/318 [00:07<00:58,  4.80it/s]\r 12%|█▏        | 38/318 [00:07<00:58,  4.79it/s]\r 12%|█▏        | 39/318 [00:08<00:57,  4.81it/s]\r 13%|█▎        | 40/318 [00:08<01:00,  4.63it/s]\r 13%|█▎        | 41/318 [00:08<00:58,  4.70it/s]\r 13%|█▎        | 42/318 [00:08<00:59,  4.68it/s]\r 14%|█▎        | 43/318 [00:08<00:56,  4.82it/s]\r 14%|█▍        | 44/318 [00:09<00:57,  4.76it/s]\r 14%|█▍        | 45/318 [00:09<00:58,  4.70it/s]\r 14%|█▍        | 46/318 [00:09<00:53,  5.09it/s]\r 15%|█▍        | 47/318 [00:09<00:53,  5.08it/s]\r 15%|█▌        | 48/318 [00:09<00:51,  5.23it/s]\r 15%|█▌        | 49/318 [00:10<00:53,  5.01it/s]\r 16%|█▌        | 50/318 [00:10<00:56,  4.72it/s]\r 16%|█▌        | 51/318 [00:10<00:56,  4.75it/s]\r 16%|█▋        | 52/318 [00:10<00:56,  4.73it/s]\r 17%|█▋        | 53/318 [00:10<00:56,  4.66it/s]\r 17%|█▋        | 54/318 [00:11<00:56,  4.65it/s]\r 17%|█▋        | 55/318 [00:11<00:56,  4.68it/s]\r 18%|█▊        | 56/318 [00:11<00:55,  4.71it/s]\r 18%|█▊        | 57/318 [00:11<00:53,  4.91it/s]\r 18%|█▊        | 58/318 [00:11<00:52,  4.93it/s]\r 19%|█▊        | 59/318 [00:12<00:51,  5.02it/s]\r 19%|█▉        | 60/318 [00:12<00:51,  5.02it/s]\r 19%|█▉        | 61/318 [00:12<00:53,  4.77it/s]\r 19%|█▉        | 62/318 [00:12<00:50,  5.03it/s]\r 20%|█▉        | 63/318 [00:12<00:48,  5.24it/s]\r 20%|██        | 64/318 [00:13<00:49,  5.11it/s]\r 20%|██        | 65/318 [00:13<00:50,  5.05it/s]\r 21%|██        | 66/318 [00:13<00:51,  4.94it/s]\r 21%|██        | 67/318 [00:13<00:51,  4.90it/s]\r 21%|██▏       | 68/318 [00:13<00:49,  5.02it/s]\r 22%|██▏       | 69/318 [00:14<00:49,  5.05it/s]\r 22%|██▏       | 70/318 [00:14<00:52,  4.77it/s]\r 22%|██▏       | 71/318 [00:14<00:50,  4.85it/s]\r 23%|██▎       | 72/318 [00:14<00:44,  5.49it/s]\r 23%|██▎       | 73/318 [00:14<00:45,  5.34it/s]\r 23%|██▎       | 74/318 [00:15<00:47,  5.17it/s]\r 24%|██▎       | 75/318 [00:15<00:47,  5.08it/s]\r 24%|██▍       | 76/318 [00:15<00:48,  4.99it/s]\r 24%|██▍       | 77/318 [00:15<00:48,  4.96it/s]\r 25%|██▍       | 78/318 [00:15<00:49,  4.88it/s]\r 25%|██▍       | 79/318 [00:16<00:48,  4.98it/s]\r 25%|██▌       | 80/318 [00:16<00:49,  4.82it/s]\r 25%|██▌       | 81/318 [00:16<00:49,  4.79it/s]\r 26%|██▌       | 82/318 [00:16<00:51,  4.59it/s]\r 26%|██▌       | 83/318 [00:16<00:48,  4.84it/s]\r 26%|██▋       | 84/318 [00:17<00:47,  4.93it/s]\r 27%|██▋       | 85/318 [00:17<00:48,  4.78it/s]\r 27%|██▋       | 86/318 [00:17<00:47,  4.86it/s]\r 27%|██▋       | 87/318 [00:17<00:48,  4.76it/s]\r 28%|██▊       | 88/318 [00:18<00:47,  4.83it/s]\r 28%|██▊       | 89/318 [00:18<00:46,  4.88it/s]\r 28%|██▊       | 90/318 [00:18<00:46,  4.94it/s]\r 29%|██▊       | 91/318 [00:18<00:47,  4.74it/s]\r 29%|██▉       | 92/318 [00:18<00:46,  4.82it/s]\r 29%|██▉       | 93/318 [00:19<00:47,  4.70it/s]\r 30%|██▉       | 94/318 [00:19<00:46,  4.81it/s]\r 30%|██▉       | 95/318 [00:19<00:45,  4.85it/s]\r 30%|███       | 96/318 [00:19<00:45,  4.90it/s]\r 31%|███       | 97/318 [00:19<00:44,  4.94it/s]\r 31%|███       | 98/318 [00:20<00:46,  4.69it/s]\r 31%|███       | 99/318 [00:20<00:47,  4.64it/s]\r 31%|███▏      | 100/318 [00:20<00:48,  4.52it/s]\r 32%|███▏      | 101/318 [00:20<00:49,  4.43it/s]\r 32%|███▏      | 102/318 [00:20<00:47,  4.53it/s]\r 32%|███▏      | 103/318 [00:21<00:46,  4.61it/s]\r 33%|███▎      | 104/318 [00:21<00:46,  4.65it/s]\r 33%|███▎      | 105/318 [00:21<00:46,  4.54it/s]\r 33%|███▎      | 106/318 [00:21<00:47,  4.44it/s]\r 34%|███▎      | 107/318 [00:22<00:45,  4.61it/s]\r 34%|███▍      | 108/318 [00:22<00:44,  4.68it/s]\r 34%|███▍      | 109/318 [00:22<00:45,  4.56it/s]\r 35%|███▍      | 110/318 [00:22<00:44,  4.66it/s]\r 35%|███▍      | 111/318 [00:22<00:44,  4.70it/s]\r 35%|███▌      | 112/318 [00:23<00:42,  4.90it/s]\r 36%|███▌      | 113/318 [00:23<00:42,  4.82it/s]\r 36%|███▌      | 114/318 [00:23<00:41,  4.88it/s]\r 36%|███▌      | 115/318 [00:23<00:43,  4.69it/s]\r 36%|███▋      | 116/318 [00:23<00:44,  4.55it/s]\r 37%|███▋      | 117/318 [00:24<00:43,  4.65it/s]\r 37%|███▋      | 118/318 [00:24<00:41,  4.79it/s]\r 37%|███▋      | 119/318 [00:24<00:40,  4.88it/s]\r 38%|███▊      | 120/318 [00:24<00:39,  4.95it/s]\r 38%|███▊      | 121/318 [00:24<00:39,  5.05it/s]\r 38%|███▊      | 122/318 [00:25<00:39,  4.98it/s]\r 39%|███▊      | 123/318 [00:25<00:39,  4.94it/s]\r 39%|███▉      | 124/318 [00:25<00:39,  4.92it/s]\r 39%|███▉      | 125/318 [00:25<00:41,  4.67it/s]\r 40%|███▉      | 126/318 [00:26<00:39,  4.86it/s]\r 40%|███▉      | 127/318 [00:26<00:38,  4.93it/s]\r 40%|████      | 128/318 [00:26<00:40,  4.70it/s]\r 41%|████      | 129/318 [00:26<00:40,  4.69it/s]\r 41%|████      | 130/318 [00:26<00:39,  4.74it/s]\r 41%|████      | 131/318 [00:27<00:36,  5.13it/s]\r 42%|████▏     | 132/318 [00:27<00:36,  5.15it/s]\r 42%|████▏     | 133/318 [00:27<00:36,  5.07it/s]\r 42%|████▏     | 134/318 [00:27<00:36,  5.08it/s]\r 42%|████▏     | 135/318 [00:27<00:38,  4.81it/s]\r 43%|████▎     | 136/318 [00:28<00:37,  4.85it/s]\r 43%|████▎     | 137/318 [00:28<00:37,  4.84it/s]\r 43%|████▎     | 138/318 [00:28<00:38,  4.63it/s]\r 44%|████▎     | 139/318 [00:28<00:37,  4.72it/s]\r 44%|████▍     | 140/318 [00:28<00:37,  4.73it/s]\r 44%|████▍     | 141/318 [00:29<00:37,  4.76it/s]\r 45%|████▍     | 142/318 [00:29<00:37,  4.74it/s]\r 45%|████▍     | 143/318 [00:29<00:35,  4.95it/s]\r 45%|████▌     | 144/318 [00:29<00:34,  5.00it/s]\r 46%|████▌     | 145/318 [00:29<00:34,  4.97it/s]\r 46%|████▌     | 146/318 [00:30<00:36,  4.70it/s]\r 46%|████▌     | 147/318 [00:30<00:34,  4.91it/s]\r 47%|████▋     | 148/318 [00:30<00:34,  4.89it/s]\r 47%|████▋     | 149/318 [00:30<00:35,  4.70it/s]\r 47%|████▋     | 150/318 [00:30<00:35,  4.69it/s]\r 47%|████▋     | 151/318 [00:31<00:35,  4.67it/s]\r 48%|████▊     | 152/318 [00:31<00:35,  4.70it/s]\r 48%|████▊     | 153/318 [00:31<00:35,  4.70it/s]\r 48%|████▊     | 154/318 [00:31<00:34,  4.70it/s]\r 49%|████▊     | 155/318 [00:32<00:34,  4.79it/s]\r 49%|████▉     | 156/318 [00:32<00:34,  4.64it/s]\r 49%|████▉     | 157/318 [00:32<00:36,  4.45it/s]\r 50%|████▉     | 158/318 [00:32<00:35,  4.54it/s]\r 50%|█████     | 159/318 [00:32<00:34,  4.61it/s]\r 50%|█████     | 160/318 [00:33<00:34,  4.62it/s]\r 51%|█████     | 161/318 [00:33<00:33,  4.65it/s]\r 51%|█████     | 162/318 [00:33<00:35,  4.44it/s]\r 51%|█████▏    | 163/318 [00:33<00:34,  4.48it/s]\r 52%|█████▏    | 164/318 [00:34<00:35,  4.34it/s]\r 52%|█████▏    | 165/318 [00:34<00:34,  4.46it/s]\r 52%|█████▏    | 166/318 [00:34<00:32,  4.62it/s]\r 53%|█████▎    | 167/318 [00:34<00:30,  4.89it/s]\r 53%|█████▎    | 168/318 [00:34<00:29,  5.04it/s]\r 53%|█████▎    | 169/318 [00:35<00:30,  4.87it/s]\r 53%|█████▎    | 170/318 [00:35<00:30,  4.85it/s]\r 54%|█████▍    | 171/318 [00:35<00:30,  4.78it/s]\r 54%|█████▍    | 172/318 [00:35<00:30,  4.79it/s]\r 54%|█████▍    | 173/318 [00:35<00:29,  4.87it/s]\r 55%|█████▍    | 174/318 [00:36<00:30,  4.68it/s]\r 55%|█████▌    | 175/318 [00:36<00:30,  4.72it/s]\r 55%|█████▌    | 176/318 [00:36<00:30,  4.71it/s]\r 56%|█████▌    | 177/318 [00:36<00:30,  4.67it/s]\r 56%|█████▌    | 178/318 [00:36<00:30,  4.66it/s]\r 56%|█████▋    | 179/318 [00:37<00:29,  4.66it/s]\r 57%|█████▋    | 180/318 [00:37<00:30,  4.55it/s]\r 57%|█████▋    | 181/318 [00:37<00:30,  4.45it/s]\r 57%|█████▋    | 182/318 [00:37<00:29,  4.54it/s]\r 58%|█████▊    | 183/318 [00:38<00:30,  4.43it/s]\r 58%|█████▊    | 184/318 [00:38<00:30,  4.39it/s]\r 58%|█████▊    | 185/318 [00:38<00:29,  4.53it/s]\r 58%|█████▊    | 186/318 [00:38<00:28,  4.65it/s]\r 59%|█████▉    | 187/318 [00:38<00:28,  4.66it/s]\r 59%|█████▉    | 188/318 [00:39<00:27,  4.76it/s]\r 59%|█████▉    | 189/318 [00:39<00:26,  4.88it/s]\r 60%|█████▉    | 190/318 [00:39<00:27,  4.73it/s]\r 60%|██████    | 191/318 [00:39<00:26,  4.84it/s]\r 60%|██████    | 192/318 [00:39<00:25,  4.99it/s]\r 61%|██████    | 193/318 [00:40<00:25,  4.96it/s]\r 61%|██████    | 194/318 [00:40<00:24,  4.98it/s]\r 61%|██████▏   | 195/318 [00:40<00:24,  5.02it/s]\r 62%|██████▏   | 196/318 [00:40<00:24,  4.99it/s]\r 62%|██████▏   | 197/318 [00:40<00:23,  5.09it/s]\r 62%|██████▏   | 198/318 [00:41<00:24,  4.81it/s]\r 63%|██████▎   | 199/318 [00:41<00:24,  4.79it/s]\r 63%|██████▎   | 200/318 [00:41<00:24,  4.88it/s]\r 63%|██████▎   | 201/318 [00:41<00:24,  4.83it/s]\r 64%|██████▎   | 202/318 [00:42<00:23,  4.87it/s]\r 64%|██████▍   | 203/318 [00:42<00:24,  4.65it/s]\r 64%|██████▍   | 204/318 [00:42<00:24,  4.65it/s]\r 64%|██████▍   | 205/318 [00:42<00:24,  4.63it/s]\r 65%|██████▍   | 206/318 [00:42<00:24,  4.49it/s]\r 65%|██████▌   | 207/318 [00:43<00:24,  4.56it/s]\r 65%|██████▌   | 208/318 [00:43<00:23,  4.64it/s]\r 66%|██████▌   | 209/318 [00:43<00:23,  4.67it/s]\r 66%|██████▌   | 210/318 [00:43<00:23,  4.67it/s]\r 66%|██████▋   | 211/318 [00:43<00:22,  4.70it/s]\r 67%|██████▋   | 212/318 [00:44<00:23,  4.52it/s]\r 67%|██████▋   | 213/318 [00:44<00:22,  4.63it/s]\r 67%|██████▋   | 214/318 [00:44<00:21,  4.78it/s]\r 68%|██████▊   | 215/318 [00:44<00:22,  4.65it/s]\r 68%|██████▊   | 216/318 [00:45<00:21,  4.83it/s]\r 68%|██████▊   | 217/318 [00:45<00:20,  4.96it/s]\r 69%|██████▊   | 218/318 [00:45<00:20,  4.90it/s]\r 69%|██████▉   | 219/318 [00:45<00:20,  4.90it/s]\r 69%|██████▉   | 220/318 [00:45<00:19,  4.96it/s]\r 69%|██████▉   | 221/318 [00:46<00:19,  5.03it/s]\r 70%|██████▉   | 222/318 [00:46<00:18,  5.05it/s]\r 70%|███████   | 223/318 [00:46<00:18,  5.03it/s]\r 70%|███████   | 224/318 [00:46<00:18,  5.06it/s]\r 71%|███████   | 225/318 [00:46<00:19,  4.86it/s]\r 71%|███████   | 226/318 [00:47<00:18,  4.93it/s]\r 71%|███████▏  | 227/318 [00:47<00:18,  4.91it/s]\r 72%|███████▏  | 228/318 [00:47<00:18,  4.96it/s]\r 72%|███████▏  | 229/318 [00:47<00:18,  4.90it/s]\r 72%|███████▏  | 230/318 [00:47<00:18,  4.77it/s]\r 73%|███████▎  | 231/318 [00:48<00:18,  4.79it/s]\r 73%|███████▎  | 232/318 [00:48<00:17,  4.93it/s]\r 73%|███████▎  | 233/318 [00:48<00:17,  4.93it/s]\r 74%|███████▎  | 234/318 [00:48<00:17,  4.81it/s]\r 74%|███████▍  | 235/318 [00:48<00:17,  4.85it/s]\r 74%|███████▍  | 236/318 [00:49<00:17,  4.68it/s]\r 75%|███████▍  | 237/318 [00:49<00:16,  4.80it/s]\r 75%|███████▍  | 238/318 [00:49<00:16,  4.89it/s]\r 75%|███████▌  | 239/318 [00:49<00:16,  4.88it/s]\r 75%|███████▌  | 240/318 [00:49<00:16,  4.64it/s]\r 76%|███████▌  | 241/318 [00:50<00:16,  4.75it/s]\r 76%|███████▌  | 242/318 [00:50<00:15,  4.89it/s]\r 76%|███████▋  | 243/318 [00:50<00:15,  4.76it/s]\r 77%|███████▋  | 244/318 [00:50<00:15,  4.74it/s]\r 77%|███████▋  | 245/318 [00:50<00:15,  4.81it/s]\r 77%|███████▋  | 246/318 [00:51<00:14,  4.84it/s]\r 78%|███████▊  | 247/318 [00:51<00:14,  4.84it/s]\r 78%|███████▊  | 248/318 [00:51<00:14,  4.85it/s]\r 78%|███████▊  | 249/318 [00:51<00:14,  4.84it/s]\r 79%|███████▊  | 250/318 [00:52<00:14,  4.70it/s]\r 79%|███████▉  | 251/318 [00:52<00:14,  4.63it/s]\r 79%|███████▉  | 252/318 [00:52<00:14,  4.71it/s]\r 80%|███████▉  | 253/318 [00:52<00:13,  4.75it/s]\r 80%|███████▉  | 254/318 [00:52<00:13,  4.76it/s]\r 80%|████████  | 255/318 [00:53<00:13,  4.59it/s]\r 81%|████████  | 256/318 [00:53<00:13,  4.70it/s]\r 81%|████████  | 257/318 [00:53<00:12,  4.93it/s]\r 81%|████████  | 258/318 [00:53<00:11,  5.07it/s]\r 81%|████████▏ | 259/318 [00:53<00:11,  5.18it/s]\r 82%|████████▏ | 260/318 [00:54<00:11,  5.10it/s]\r 82%|████████▏ | 261/318 [00:54<00:11,  5.03it/s]\r 82%|████████▏ | 262/318 [00:54<00:11,  5.09it/s]\r 83%|████████▎ | 263/318 [00:54<00:10,  5.15it/s]\r 83%|████████▎ | 264/318 [00:54<00:10,  5.07it/s]\r 83%|████████▎ | 265/318 [00:55<00:10,  5.07it/s]\r 84%|████████▎ | 266/318 [00:55<00:10,  4.99it/s]\r 84%|████████▍ | 267/318 [00:55<00:09,  5.14it/s]\r 84%|████████▍ | 268/318 [00:55<00:09,  5.22it/s]\r 85%|████████▍ | 269/318 [00:55<00:09,  5.25it/s]\r 85%|████████▍ | 270/318 [00:55<00:09,  5.32it/s]\r 85%|████████▌ | 271/318 [00:56<00:08,  5.33it/s]\r 86%|████████▌ | 272/318 [00:56<00:08,  5.17it/s]\r 86%|████████▌ | 273/318 [00:56<00:08,  5.22it/s]\r 86%|████████▌ | 274/318 [00:56<00:08,  5.06it/s]\r 86%|████████▋ | 275/318 [00:56<00:08,  5.09it/s]\r 87%|████████▋ | 276/318 [00:57<00:08,  5.09it/s]\r 87%|████████▋ | 277/318 [00:57<00:08,  5.01it/s]\r 87%|████████▋ | 278/318 [00:57<00:08,  4.95it/s]\r 88%|████████▊ | 279/318 [00:57<00:07,  4.95it/s]\r 88%|████████▊ | 280/318 [00:58<00:07,  4.94it/s]\r 88%|████████▊ | 281/318 [00:58<00:07,  4.95it/s]\r 89%|████████▊ | 282/318 [00:58<00:07,  5.00it/s]\r 89%|████████▉ | 283/318 [00:58<00:07,  4.86it/s]\r 89%|████████▉ | 284/318 [00:58<00:07,  4.80it/s]\r 90%|████████▉ | 285/318 [00:59<00:06,  4.76it/s]\r 90%|████████▉ | 286/318 [00:59<00:06,  4.64it/s]\r 90%|█████████ | 287/318 [00:59<00:06,  4.69it/s]\r 91%|█████████ | 288/318 [00:59<00:06,  4.87it/s]\r 91%|█████████ | 289/318 [00:59<00:06,  4.79it/s]\r 91%|█████████ | 290/318 [01:00<00:05,  4.87it/s]\r 92%|█████████▏| 291/318 [01:00<00:05,  4.96it/s]\r 92%|█████████▏| 292/318 [01:00<00:05,  4.91it/s]\r 92%|█████████▏| 293/318 [01:00<00:04,  5.05it/s]\r 92%|█████████▏| 294/318 [01:00<00:04,  5.12it/s]\r 93%|█████████▎| 295/318 [01:01<00:04,  5.10it/s]\r 93%|█████████▎| 296/318 [01:01<00:04,  5.09it/s]\r 93%|█████████▎| 297/318 [01:01<00:04,  5.07it/s]\r 94%|█████████▎| 298/318 [01:01<00:04,  4.99it/s]\r 94%|█████████▍| 299/318 [01:01<00:03,  4.96it/s]\r 94%|█████████▍| 300/318 [01:02<00:03,  4.87it/s]\r 95%|█████████▍| 301/318 [01:02<00:03,  4.93it/s]\r 95%|█████████▍| 302/318 [01:02<00:03,  4.96it/s]\r 95%|█████████▌| 303/318 [01:02<00:03,  4.83it/s]\r 96%|█████████▌| 304/318 [01:02<00:02,  4.75it/s]\r 96%|█████████▌| 305/318 [01:03<00:02,  4.88it/s]\r 96%|█████████▌| 306/318 [01:03<00:02,  4.98it/s]\r 97%|█████████▋| 307/318 [01:03<00:02,  5.14it/s]\r 97%|█████████▋| 308/318 [01:03<00:01,  5.29it/s]\r 97%|█████████▋| 309/318 [01:03<00:01,  5.34it/s]\r 97%|█████████▋| 310/318 [01:04<00:01,  5.35it/s]\r 98%|█████████▊| 311/318 [01:04<00:01,  5.19it/s]\r 98%|█████████▊| 312/318 [01:04<00:01,  5.19it/s]\r 98%|█████████▊| 313/318 [01:04<00:00,  5.25it/s]\r 99%|█████████▊| 314/318 [01:04<00:00,  5.23it/s]\r 99%|█████████▉| 315/318 [01:04<00:00,  5.32it/s]\r 99%|█████████▉| 316/318 [01:05<00:00,  5.38it/s]\r100%|█████████▉| 317/318 [01:05<00:00,  5.36it/s]\r100%|██████████| 318/318 [01:05<00:00,  6.03it/s]\r100%|██████████| 318/318 [01:05<00:00,  4.86it/s]
Traceback (most recent call last):
  File \"/Users/aghat/Documents/analysis/chenlab_organoids/Dec14_2023/in_container_sorter_script.py\", line 23, in <module>
drift computed in  68.67s; total  71.52s

Extracting spikes using templates
Re-computing universal templates from data.
Error running kilosort4
    sorting = run_sorter_local(
              ^^^^^^^^^^^^^^^^^
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py\", line 225, in run_sorter_local
    SorterClass.run_from_folder(output_folder, raise_error, verbose)
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py\", line 293, in run_from_folder
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py\", line 258, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File \"/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/external/kilosort4.py\", line 260, in _run_from_folder
    st, tF, _, _ = detect_spikes(ops, device, bfile, tic0=tic0, progress_bar=progress_bar)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/kilosort/run_kilosort.py\", line 392, in detect_spikes
    st0, tF, ops = spikedetect.run(ops, bfile, device=device, progress_bar=progress_bar)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py\", line 193, in run
    ops['wPCA'], ops['wTEMP'] = extract_wPCA_wTEMP(
                                ^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py\", line 70, in extract_wPCA_wTEMP
    model = TruncatedSVD(n_components=ops['settings']['n_pcs']).fit(clips)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/decomposition/_truncated_svd.py\", line 209, in fit
    self.fit_transform(X)
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/utils/_set_output.py\", line 295, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/base.py\", line 1474, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/decomposition/_truncated_svd.py\", line 229, in fit_transform
    X = self._validate_data(X, accept_sparse=[\"csr\", \"csc\"], ensure_min_features=2)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/base.py\", line 633, in _validate_data
    out = check_array(X, input_name=\"X\", **check_params)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/home/miniconda3/lib/python3.11/site-packages/sklearn/utils/validation.py\", line 1072, in check_array
    raise ValueError(
ValueError: Found array with 0 sample(s) (shape=(0, 61)) while a minimum of 1 is required by TruncatedSVD.

Spike sorting failed. You can inspect the runtime trace in /Users/aghat/Documents/analysis/chenlab_organoids/Dec14_2023/results__nopre_rec1_2_KS4_0drift/spikeinterface_log.json.
"
}

[alejoe91]

Hi @aghatpande

The warning warn("There is no Probe attached to this recording. Creating a dummy one with contact positions") is ok. the reason you're seeing it is because when "passing" the recording to the docker container, only the channels locations (and groups) are passed, not the entire probe. Channel locations are enough for spike sorting.

The Kilosort4 error is unrelated to this and it seems to be related to the fact that no spikes are detected in a batch. Can you try to increase the batch_size?

You can add it to the run_sorter() function: run_sorter(..., batch_size=120000) (by default it's 60000 ~ 2s)

[aghatpande]

Thanks for your suggestion. I did try the 120000 for batch_size which still resulted in no spikes (this same data gave 120 units using KS3). I have played around with several parameters (batch_size, nblocks, Th_universal, nearest_chans and nearest_templates) none leads to even a single spike being detected!. My probe is a multi-shank probe. Kilosort4 documentation suggests that it had issues with these types of probes but has been fixed in the latest version of their code. How do I get SI to run the latest version of Kilosort4 as a last-ditch try? Anyway thank you for all the help so far. I feel like I can live with using KS3 for my data as it is.

[chrishalcrow]

Hello, we had issues with the latest versions of kilosort and downgraded to version 4.0.6 (pip install kilosort==4.0.6 if you're pip'ing). I have 10% confidence that it'll work for you, but could be worth a try!

[aghatpande]

thanks @chrishalcrow for the suggestion. I'm pausing my KS4 attempts for now.

[mberns-ru]

Hi! I wanted to report that I've been having the exact same issue (error says the exact same thing). Changing the batch size to 120k and installing an old version of KS4 didn't work for me either. I'll also switch away from KS4 for the time being.