search

SpikeInterface / spikeinterface

A Python-based module for creating flexible and robust spike sorting pipelines.
https://spikeinterface.readthedocs.io
MIT License
526 stars 187 forks source link

kilosort4 runtime error with skiping the drift correction #2699

Closed Sachuriga closed 7 months ago

Sachuriga commented 7 months ago

Dear,

I'm testing on the kilosrt4 with 6shank 64 channel probe data. When I skipping the drift correction the kilosor4 stopped with error:

Skipping kilosort preprocessing.
Skipping drift correction.

'Extracting spikes using templates
Re-computing universal templates from data.
Traceback (most recent call last):
  File "/cluster/home/sachur/Overlap_project/code/Python/Kilosort/kilosort4_multfile.py", line 188, in <module>
    sorting = ss.run_sorter_by_property('kilosort4',
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/launcher.py", line 297, in run_sorter_by_property
    sorting_list = run_sorter_jobs(job_list, engine=engine, engine_kwargs=engine_kwargs, return_output=True)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/launcher.py", line 106, in run_sorter_jobs
    sorting = run_sorter(**kwargs)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/runsorter.py", line 175, in run_sorter
    return run_sorter_local(**common_kwargs)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/runsorter.py", line 225, in run_sorter_local
    SorterClass.run_from_folder(output_folder, raise_error, verbose)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/basesorter.py", line 293, in run_from_folder
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/basesorter.py", line 258, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/external/kilosort4.py", line 260, in _run_from_folder
    st, tF, _, _ = detect_spikes(ops, device, bfile, tic0=tic0, progress_bar=progress_bar)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/run_kilosort.py", line 392, in detect_spikes
    st0, tF, ops = spikedetect.run(ops, bfile, device=device, progress_bar=progress_bar)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/spikedetect.py", line 193, in run
    ops['wPCA'], ops['wTEMP'] = extract_wPCA_wTEMP(
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/spikedetect.py", line 54, in extract_wPCA_wTEMP
    X = bfile.padded_batch_to_torch(j, ops)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/io.py", line 666, in padded_batch_to_torch
    X = super().padded_batch_to_torch(ibatch)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/io.py", line 494, in padded_batch_to_torch
    X = torch.zeros((self.n_chan_bin, self.NT + 2*self.nt), device=self.device)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.'

And the paras for sorter is like below,

para_ks4 = {'skip_kilosort_preprocessing': True,
                    'nblocks': 0,
                    'nearest_templates': 9,
                    'min_template_size': 16.5,
                    'batch_size': 320000,
                    'do_correction':False}

sorting = ss.run_sorter_by_property('kilosort4',
                                    recording=recording_f0,
                                    working_folder=base_folder / ks3,
                                    grouping_property='group',
                                    mode_if_folder_exists='keep', **para_ks4)

Would you think this could be a error from spikeinterface or kilosort4?

best,

zm711 commented 7 months ago

What happens if you type 

import torch

print(torch.cuda.is_available())

A cuda error like this is likely KS4 because their code is where the cuda calls are made.

Sachuriga commented 7 months ago

Hey I have tested the code, its giving me True .

and if I set the nblock=1 and do_correction=True the sorter is works.

Loading recording with SpikeInterface...
number of samples: 38698575
number of channels: 10
numbef of segments: 1
sampling rate: 30000.0
dtype: int16
========================================
Skipping kilosort preprocessing.

computing drift
Re-computing universal templates from data.

  0%|          | 0/121 [00:00<?, ?it/s]
  1%|          | 1/121 [00:03<06:16,  3.14s/it]
  2%|▏         | 2/121 [00:04<03:59,  2.02s/it]
  2%|▏         | 3/121 [00:05<03:15,  1.66s/it]
  3%|▎         | 4/121 [00:06<02:54,  1.49s/it]
  4%|▍         | 5/121 [00:08<02:41,  1.40s/it]
  5%|▍         | 6/121 [00:09<02:44,  1.43s/it]
  6%|▌         | 7/121 [00:10<02:35,  1.36s/it]
  7%|▋         | 8/121 [00:12<02:29,  1.32s/it]
  7%|▋         | 9/121 [00:13<02:24,  1.29s/it]
  8%|▊         | 10/121 [00:14<02:20,  1.27s/it]
  9%|▉         | 11/121 [00:15<02:18,  1.26s/it]
 10%|▉         | 12/121 [00:17<02:34,  1.41s/it]
 11%|█         | 13/121 [00:18<02:26,  1.36s/it]
 12%|█▏        | 14/121 [00:19<02:20,  1.32s/it]
 12%|█▏        | 15/121 [00:21<02:16,  1.29s/it]
 13%|█▎        | 16/121 [00:22<02:13,  1.27s/it]
 14%|█▍        | 17/121 [00:23<02:10,  1.26s/it]
 15%|█▍        | 18/121 [00:24<02:08,  1.25s/it]
 16%|█▌        | 19/121 [00:26<02:06,  1.24s/it]
 17%|█▋        | 20/121 [00:27<02:04,  1.23s/it]
 17%|█▋        | 21/121 [00:28<02:03,  1.23s/it]
 18%|█▊        | 22/121 [00:29<02:01,  1.23s/it]
 19%|█▉        | 23/121 [00:32<02:32,  1.56s/it]
 20%|█▉        | 24/121 [00:33<02:21,  1.46s/it]
 21%|██        | 25/121 [00:34<02:13,  1.39s/it]
 21%|██▏       | 26/121 [00:35<02:06,  1.33s/it]
 22%|██▏       | 27/121 [00:36<02:02,  1.30s/it]
 23%|██▎       | 28/121 [00:38<01:58,  1.28s/it]
 24%|██▍       | 29/121 [00:39<01:56,  1.26s/it]
 25%|██▍       | 30/121 [00:40<01:53,  1.25s/it]
 26%|██▌       | 31/121 [00:41<01:51,  1.24s/it]
 26%|██▋       | 32/121 [00:43<01:50,  1.24s/it]
 27%|██▋       | 33/121 [00:44<01:48,  1.23s/it]
 28%|██▊       | 34/121 [00:45<01:47,  1.23s/it]
 29%|██▉       | 35/121 [00:46<01:45,  1.23s/it]
 30%|██▉       | 36/121 [00:47<01:44,  1.23s/it]
 31%|███       | 37/121 [00:49<01:42,  1.23s/it]
 31%|███▏      | 38/121 [00:50<01:41,  1.23s/it]
 32%|███▏      | 39/121 [00:51<01:40,  1.23s/it]
 33%|███▎      | 40/121 [00:52<01:39,  1.22s/it]
 34%|███▍      | 41/121 [00:54<01:37,  1.22s/it]
 35%|███▍      | 42/121 [00:55<01:36,  1.22s/it]
 36%|███▌      | 43/121 [00:56<01:35,  1.22s/it]
 36%|███▋      | 44/121 [00:57<01:34,  1.22s/it]
 37%|███▋      | 45/121 [01:01<02:23,  1.88s/it]
 38%|███▊      | 46/121 [01:02<02:06,  1.69s/it]
 39%|███▉      | 47/121 [01:03<01:54,  1.55s/it]
 40%|███▉      | 48/121 [01:04<01:45,  1.45s/it]
 40%|████      | 49/121 [01:06<01:39,  1.38s/it]
 41%|████▏     | 50/121 [01:07<01:34,  1.33s/it]
 42%|████▏     | 51/121 [01:08<01:30,  1.30s/it]
 43%|████▎     | 52/121 [01:09<01:27,  1.27s/it]
 44%|████▍     | 53/121 [01:10<01:25,  1.26s/it]
 45%|████▍     | 54/121 [01:12<01:23,  1.25s/it]
 45%|████▌     | 55/121 [01:13<01:21,  1.24s/it]
 46%|████▋     | 56/121 [01:14<01:20,  1.23s/it]
 47%|████▋     | 57/121 [01:15<01:18,  1.23s/it]
 48%|████▊     | 58/121 [01:17<01:17,  1.23s/it]
 49%|████▉     | 59/121 [01:18<01:16,  1.23s/it]
 50%|████▉     | 60/121 [01:19<01:14,  1.22s/it]
 50%|█████     | 61/121 [01:20<01:13,  1.22s/it]
 51%|█████     | 62/121 [01:21<01:12,  1.22s/it]
 52%|█████▏    | 63/121 [01:23<01:10,  1.22s/it]
 53%|█████▎    | 64/121 [01:24<01:09,  1.22s/it]
 54%|█████▎    | 65/121 [01:25<01:08,  1.22s/it]
 55%|█████▍    | 66/121 [01:26<01:07,  1.22s/it]
 55%|█████▌    | 67/121 [01:28<01:06,  1.22s/it]
 56%|█████▌    | 68/121 [01:29<01:04,  1.22s/it]
 57%|█████▋    | 69/121 [01:30<01:03,  1.22s/it]
 58%|█████▊    | 70/121 [01:31<01:02,  1.22s/it]
 59%|█████▊    | 71/121 [01:32<01:01,  1.22s/it]
 60%|█████▉    | 72/121 [01:34<00:59,  1.22s/it]
 60%|██████    | 73/121 [01:35<00:58,  1.22s/it]
zm711 commented 7 months ago

Could you try your original code with do_correction=False and with nblocks=1 and let us know if that works?

Sachuriga commented 7 months ago

Hey,

the log file above is generated with my code like this, and I have added the code you proivided thats showing True in the log file.

para_ks4 = {'skip_kilosort_preprocessing': True,
                    'nblocks': 1,
                    'nearest_templates': 9,
                    'min_template_size': 16.5,
                    'batch_size': 320000,
                    'do_correction':True}

sorting = ss.run_sorter_by_property('kilosort4',
                                    recording=recording_f0,
                                    working_folder=base_folder / ks3,
                                    grouping_property='group',
                                    mode_if_folder_exists='keep', **para_ks4)

the full log will be looks like this, there happens a another error...but at least it step to the clustering process..

Wed Apr 10 20:04:54 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.06              Driver Version: 545.23.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P100-PCIE-16GB           On  | 00000000:03:00.0 Off |                    0 |
| N/A   39C    P0              36W / 250W |      0MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
/cluster/apps/eb/software/Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
True
Current working directory before
/cluster/home/sachur/Overlap_project/data/OpenEphys

65588_2024-03-08
65588_2024-03-08_16-10-29_C
Current working directory before
/cluster/home/sachur/Overlap_project/data/OpenEphys

Channel ids: ['CH1' 'CH2' 'CH3' 'CH4' 'CH5' 'CH6' 'CH7' 'CH8' 'CH9' 'CH10' 'CH11'
 'CH12' 'CH13' 'CH14' 'CH15' 'CH16' 'CH17' 'CH18' 'CH19' 'CH20' 'CH21'
 'CH22' 'CH23' 'CH24' 'CH25' 'CH26' 'CH27' 'CH28' 'CH29' 'CH30' 'CH31'
 'CH32' 'CH33' 'CH34' 'CH35' 'CH36' 'CH37' 'CH38' 'CH39' 'CH40' 'CH41'
 'CH42' 'CH43' 'CH44' 'CH45' 'CH46' 'CH47' 'CH48' 'CH49' 'CH50' 'CH51'
 'CH52' 'CH53' 'CH54' 'CH55' 'CH56' 'CH57' 'CH58' 'CH59' 'CH60' 'CH61'
 'CH62' 'CH63' 'CH64']
Sampling frequency: 30000.0
Number of channels: 64
Number of segments: 1
Traces shape: (60000, 64)
ASSY-236-F - cambridgeneurotech - 64ch - 6shanks
['is_filtered', 'probe_0_planar_contour']
write_binary_recording with n_jobs = 40 and chunk_size = 150000

write_binary_recording:   0%|          | 0/258 [00:00<?, ?it/s]
write_binary_recording:   0%|          | 1/258 [00:05<24:57,  5.83s/it]
write_binary_recording:   1%|          | 3/258 [00:06<07:29,  1.76s/it]
write_binary_recording:   5%|▍         | 12/258 [00:07<01:32,  2.65it/s]
write_binary_recording:  10%|█         | 26/258 [00:07<00:36,  6.42it/s]
write_binary_recording:  12%|█▏        | 30/258 [00:07<00:29,  7.62it/s]
write_binary_recording:  12%|█▏        | 32/258 [00:08<00:31,  7.15it/s]
write_binary_recording:  13%|█▎        | 34/258 [00:08<00:32,  6.82it/s]
write_binary_recording:  15%|█▌        | 39/258 [00:09<00:27,  7.89it/s]
write_binary_recording:  16%|█▌        | 41/258 [00:11<01:02,  3.47it/s]
write_binary_recording:  16%|█▋        | 42/258 [00:12<01:21,  2.67it/s]
write_binary_recording:  17%|█▋        | 44/258 [00:12<01:14,  2.87it/s]
write_binary_recording:  19%|█▊        | 48/258 [00:13<00:47,  4.38it/s]
write_binary_recording:  19%|█▉        | 49/258 [00:13<00:56,  3.71it/s]
write_binary_recording:  19%|█▉        | 50/258 [00:14<01:01,  3.39it/s]
write_binary_recording:  20%|██        | 52/258 [00:14<01:01,  3.33it/s]
write_binary_recording:  21%|██        | 54/258 [00:15<00:54,  3.72it/s]
write_binary_recording:  22%|██▏       | 58/258 [00:15<00:44,  4.48it/s]
write_binary_recording:  24%|██▎       | 61/258 [00:17<00:55,  3.52it/s]
write_binary_recording:  31%|███▏      | 81/258 [00:18<00:19,  9.20it/s]
write_binary_recording:  32%|███▏      | 82/258 [00:19<00:29,  6.03it/s]
write_binary_recording:  32%|███▏      | 83/258 [00:20<00:37,  4.65it/s]
write_binary_recording:  35%|███▌      | 91/258 [00:20<00:23,  7.24it/s]
write_binary_recording:  36%|███▌      | 93/258 [00:22<00:38,  4.28it/s]
write_binary_recording:  36%|███▋      | 94/258 [00:23<00:54,  3.03it/s]
write_binary_recording:  38%|███▊      | 97/258 [00:23<00:41,  3.87it/s]
write_binary_recording:  41%|████▏     | 107/258 [00:24<00:23,  6.34it/s]
write_binary_recording:  43%|████▎     | 111/258 [00:24<00:19,  7.43it/s]
write_binary_recording:  44%|████▍     | 113/258 [00:25<00:19,  7.28it/s]
write_binary_recording:  45%|████▍     | 115/258 [00:26<00:27,  5.13it/s]
write_binary_recording:  45%|████▌     | 117/258 [00:26<00:23,  5.94it/s]
write_binary_recording:  48%|████▊     | 124/258 [00:26<00:16,  8.08it/s]
write_binary_recording:  49%|████▉     | 126/258 [00:27<00:19,  6.70it/s]
write_binary_recording:  50%|████▉     | 128/258 [00:27<00:17,  7.43it/s]
write_binary_recording:  50%|█████     | 130/258 [00:27<00:17,  7.15it/s]
write_binary_recording:  52%|█████▏    | 133/258 [00:28<00:19,  6.39it/s]
write_binary_recording:  52%|█████▏    | 134/258 [00:28<00:27,  4.55it/s]
write_binary_recording:  53%|█████▎    | 137/258 [00:29<00:26,  4.60it/s]
write_binary_recording:  53%|█████▎    | 138/258 [00:29<00:25,  4.76it/s]
write_binary_recording:  54%|█████▍    | 139/258 [00:29<00:23,  5.01it/s]
write_binary_recording:  54%|█████▍    | 140/258 [00:30<00:25,  4.64it/s]
write_binary_recording:  55%|█████▌    | 143/258 [00:31<00:33,  3.38it/s]
write_binary_recording:  56%|█████▌    | 144/258 [00:31<00:30,  3.78it/s]
write_binary_recording:  57%|█████▋    | 146/258 [00:31<00:27,  4.13it/s]
write_binary_recording:  59%|█████▊    | 151/258 [00:32<00:13,  7.98it/s]
write_binary_recording:  60%|█████▉    | 154/258 [00:32<00:15,  6.91it/s]
write_binary_recording:  60%|██████    | 156/258 [00:33<00:18,  5.49it/s]
write_binary_recording:  61%|██████    | 158/258 [00:33<00:15,  6.28it/s]
write_binary_recording:  62%|██████▏   | 159/258 [00:33<00:16,  6.00it/s]
write_binary_recording:  62%|██████▏   | 160/258 [00:33<00:17,  5.76it/s]
write_binary_recording:  63%|██████▎   | 162/258 [00:33<00:14,  6.65it/s]
write_binary_recording:  63%|██████▎   | 163/258 [00:34<00:15,  6.03it/s]
write_binary_recording:  64%|██████▎   | 164/258 [00:34<00:15,  6.12it/s]
write_binary_recording:  65%|██████▌   | 168/258 [00:35<00:15,  5.93it/s]
write_binary_recording:  67%|██████▋   | 172/258 [00:35<00:09,  9.06it/s]
write_binary_recording:  67%|██████▋   | 174/258 [00:35<00:12,  6.65it/s]
write_binary_recording:  68%|██████▊   | 175/258 [00:35<00:12,  6.74it/s]
write_binary_recording:  68%|██████▊   | 176/258 [00:36<00:12,  6.75it/s]
write_binary_recording:  69%|██████▊   | 177/258 [00:36<00:19,  4.22it/s]
write_binary_recording:  69%|██████▉   | 179/258 [00:37<00:17,  4.54it/s]
write_binary_recording:  70%|███████   | 181/258 [00:37<00:15,  5.03it/s]
write_binary_recording:  71%|███████   | 183/258 [00:38<00:21,  3.47it/s]
write_binary_recording:  71%|███████▏  | 184/258 [00:38<00:19,  3.86it/s]
write_binary_recording:  72%|███████▏  | 186/258 [00:38<00:15,  4.53it/s]
write_binary_recording:  73%|███████▎  | 188/258 [00:39<00:20,  3.35it/s]
write_binary_recording:  74%|███████▎  | 190/258 [00:39<00:15,  4.43it/s]
write_binary_recording:  76%|███████▌  | 196/258 [00:40<00:10,  6.10it/s]
write_binary_recording:  76%|███████▋  | 197/258 [00:40<00:10,  6.02it/s]
write_binary_recording:  77%|███████▋  | 198/258 [00:40<00:09,  6.26it/s]
write_binary_recording:  78%|███████▊  | 201/258 [00:40<00:06,  8.92it/s]
write_binary_recording:  79%|███████▊  | 203/258 [00:41<00:06,  8.82it/s]
write_binary_recording:  79%|███████▉  | 205/258 [00:42<00:12,  4.35it/s]
write_binary_recording:  81%|████████  | 208/258 [00:42<00:09,  5.28it/s]
write_binary_recording:  81%|████████▏ | 210/258 [00:43<00:09,  4.98it/s]
write_binary_recording:  83%|████████▎ | 213/258 [00:43<00:10,  4.50it/s]
write_binary_recording:  84%|████████▍ | 217/258 [00:44<00:08,  4.83it/s]
write_binary_recording:  84%|████████▍ | 218/258 [00:44<00:08,  4.57it/s]
write_binary_recording:  85%|████████▍ | 219/258 [00:45<00:09,  4.00it/s]
write_binary_recording:  87%|████████▋ | 225/258 [00:45<00:04,  7.95it/s]
write_binary_recording:  88%|████████▊ | 228/258 [00:45<00:03,  7.58it/s]
write_binary_recording:  89%|████████▉ | 230/258 [00:46<00:05,  4.76it/s]
write_binary_recording:  92%|█████████▏| 237/258 [00:47<00:03,  6.08it/s]
write_binary_recording:  93%|█████████▎| 239/258 [00:47<00:02,  6.72it/s]
write_binary_recording:  94%|█████████▍| 242/258 [00:48<00:01,  8.12it/s]
write_binary_recording:  95%|█████████▌| 246/258 [00:48<00:01, 11.04it/s]
write_binary_recording: 100%|██████████| 258/258 [00:48<00:00, 23.60it/s]
write_binary_recording: 100%|██████████| 258/258 [00:48<00:00,  5.34it/s]
Setting IRONCLUST_PATH environment variable for subprocess calls to: /cluster/home/sachur/Overlap_project/code/Matlab/ironclust-master
========================================
Loading recording with SpikeInterface...
number of samples: 38698575
number of channels: 10
numbef of segments: 1
sampling rate: 30000.0
dtype: int16
========================================
Skipping kilosort preprocessing.

computing drift
Re-computing universal templates from data.

  0%|          | 0/121 [00:00<?, ?it/s]
  1%|          | 1/121 [00:03<06:16,  3.14s/it]
  2%|▏         | 2/121 [00:04<03:59,  2.02s/it]
  2%|▏         | 3/121 [00:05<03:15,  1.66s/it]
  3%|▎         | 4/121 [00:06<02:54,  1.49s/it]
  4%|▍         | 5/121 [00:08<02:41,  1.40s/it]
  5%|▍         | 6/121 [00:09<02:44,  1.43s/it]
  6%|▌         | 7/121 [00:10<02:35,  1.36s/it]
  7%|▋         | 8/121 [00:12<02:29,  1.32s/it]
  7%|▋         | 9/121 [00:13<02:24,  1.29s/it]
  8%|▊         | 10/121 [00:14<02:20,  1.27s/it]
  9%|▉         | 11/121 [00:15<02:18,  1.26s/it]
 10%|▉         | 12/121 [00:17<02:34,  1.41s/it]
 11%|█         | 13/121 [00:18<02:26,  1.36s/it]
 12%|█▏        | 14/121 [00:19<02:20,  1.32s/it]
 12%|█▏        | 15/121 [00:21<02:16,  1.29s/it]
 13%|█▎        | 16/121 [00:22<02:13,  1.27s/it]
 14%|█▍        | 17/121 [00:23<02:10,  1.26s/it]
 15%|█▍        | 18/121 [00:24<02:08,  1.25s/it]
 16%|█▌        | 19/121 [00:26<02:06,  1.24s/it]
 17%|█▋        | 20/121 [00:27<02:04,  1.23s/it]
 17%|█▋        | 21/121 [00:28<02:03,  1.23s/it]
 18%|█▊        | 22/121 [00:29<02:01,  1.23s/it]
 19%|█▉        | 23/121 [00:32<02:32,  1.56s/it]
 20%|█▉        | 24/121 [00:33<02:21,  1.46s/it]
 21%|██        | 25/121 [00:34<02:13,  1.39s/it]
 21%|██▏       | 26/121 [00:35<02:06,  1.33s/it]
 22%|██▏       | 27/121 [00:36<02:02,  1.30s/it]
 23%|██▎       | 28/121 [00:38<01:58,  1.28s/it]
 24%|██▍       | 29/121 [00:39<01:56,  1.26s/it]
 25%|██▍       | 30/121 [00:40<01:53,  1.25s/it]
 26%|██▌       | 31/121 [00:41<01:51,  1.24s/it]
 26%|██▋       | 32/121 [00:43<01:50,  1.24s/it]
 27%|██▋       | 33/121 [00:44<01:48,  1.23s/it]
 28%|██▊       | 34/121 [00:45<01:47,  1.23s/it]
 29%|██▉       | 35/121 [00:46<01:45,  1.23s/it]
 30%|██▉       | 36/121 [00:47<01:44,  1.23s/it]
 31%|███       | 37/121 [00:49<01:42,  1.23s/it]
 31%|███▏      | 38/121 [00:50<01:41,  1.23s/it]
 32%|███▏      | 39/121 [00:51<01:40,  1.23s/it]
 33%|███▎      | 40/121 [00:52<01:39,  1.22s/it]
 34%|███▍      | 41/121 [00:54<01:37,  1.22s/it]
 35%|███▍      | 42/121 [00:55<01:36,  1.22s/it]
 36%|███▌      | 43/121 [00:56<01:35,  1.22s/it]
 36%|███▋      | 44/121 [00:57<01:34,  1.22s/it]
 37%|███▋      | 45/121 [01:01<02:23,  1.88s/it]
 38%|███▊      | 46/121 [01:02<02:06,  1.69s/it]
 39%|███▉      | 47/121 [01:03<01:54,  1.55s/it]
 40%|███▉      | 48/121 [01:04<01:45,  1.45s/it]
 40%|████      | 49/121 [01:06<01:39,  1.38s/it]
 41%|████▏     | 50/121 [01:07<01:34,  1.33s/it]
 42%|████▏     | 51/121 [01:08<01:30,  1.30s/it]
 43%|████▎     | 52/121 [01:09<01:27,  1.27s/it]
 44%|████▍     | 53/121 [01:10<01:25,  1.26s/it]
 45%|████▍     | 54/121 [01:12<01:23,  1.25s/it]
 45%|████▌     | 55/121 [01:13<01:21,  1.24s/it]
 46%|████▋     | 56/121 [01:14<01:20,  1.23s/it]
 47%|████▋     | 57/121 [01:15<01:18,  1.23s/it]
 48%|████▊     | 58/121 [01:17<01:17,  1.23s/it]
 49%|████▉     | 59/121 [01:18<01:16,  1.23s/it]
 50%|████▉     | 60/121 [01:19<01:14,  1.22s/it]
 50%|█████     | 61/121 [01:20<01:13,  1.22s/it]
 51%|█████     | 62/121 [01:21<01:12,  1.22s/it]
 52%|█████▏    | 63/121 [01:23<01:10,  1.22s/it]
 53%|█████▎    | 64/121 [01:24<01:09,  1.22s/it]
 54%|█████▎    | 65/121 [01:25<01:08,  1.22s/it]
 55%|█████▍    | 66/121 [01:26<01:07,  1.22s/it]
 55%|█████▌    | 67/121 [01:28<01:06,  1.22s/it]
 56%|█████▌    | 68/121 [01:29<01:04,  1.22s/it]
 57%|█████▋    | 69/121 [01:30<01:03,  1.22s/it]
 58%|█████▊    | 70/121 [01:31<01:02,  1.22s/it]
 59%|█████▊    | 71/121 [01:32<01:01,  1.22s/it]
 60%|█████▉    | 72/121 [01:34<00:59,  1.22s/it]
 60%|██████    | 73/121 [01:35<00:58,  1.22s/it]
 61%|██████    | 74/121 [01:36<00:57,  1.22s/it]
 62%|██████▏   | 75/121 [01:37<00:56,  1.22s/it]
 63%|██████▎   | 76/121 [01:38<00:54,  1.22s/it]
 64%|██████▎   | 77/121 [01:40<00:53,  1.22s/it]
 64%|██████▍   | 78/121 [01:41<00:52,  1.22s/it]
 65%|██████▌   | 79/121 [01:42<00:51,  1.22s/it]
 66%|██████▌   | 80/121 [01:43<00:50,  1.22s/it]
 67%|██████▋   | 81/121 [01:45<00:48,  1.22s/it]
 68%|██████▊   | 82/121 [01:46<00:47,  1.22s/it]
 69%|██████▊   | 83/121 [01:47<00:46,  1.22s/it]
 69%|██████▉   | 84/121 [01:48<00:45,  1.22s/it]
 70%|███████   | 85/121 [01:49<00:44,  1.22s/it]
 71%|███████   | 86/121 [01:51<00:42,  1.22s/it]
 72%|███████▏  | 87/121 [01:52<00:41,  1.22s/it]
 73%|███████▎  | 88/121 [01:53<00:40,  1.22s/it]
 74%|███████▎  | 89/121 [01:54<00:39,  1.22s/it]
 74%|███████▍  | 90/121 [02:00<01:18,  2.54s/it]
 75%|███████▌  | 91/121 [02:01<01:04,  2.15s/it]
 76%|███████▌  | 92/121 [02:02<00:54,  1.87s/it]
 77%|███████▋  | 93/121 [02:04<00:46,  1.68s/it]
 78%|███████▊  | 94/121 [02:05<00:41,  1.54s/it]
 79%|███████▊  | 95/121 [02:06<00:37,  1.45s/it]
 79%|███████▉  | 96/121 [02:07<00:34,  1.38s/it]
 80%|████████  | 97/121 [02:09<00:31,  1.33s/it]
 81%|████████  | 98/121 [02:10<00:29,  1.30s/it]
 82%|████████▏ | 99/121 [02:11<00:28,  1.28s/it]
 83%|████████▎ | 100/121 [02:12<00:26,  1.26s/it]
 83%|████████▎ | 101/121 [02:13<00:24,  1.25s/it]
 84%|████████▍ | 102/121 [02:15<00:23,  1.24s/it]
 85%|████████▌ | 103/121 [02:16<00:22,  1.24s/it]
 86%|████████▌ | 104/121 [02:17<00:20,  1.23s/it]
 87%|████████▋ | 105/121 [02:18<00:19,  1.23s/it]
 88%|████████▊ | 106/121 [02:20<00:18,  1.23s/it]
 88%|████████▊ | 107/121 [02:21<00:17,  1.23s/it]
 89%|████████▉ | 108/121 [02:22<00:15,  1.23s/it]
 90%|█████████ | 109/121 [02:23<00:14,  1.22s/it]
 91%|█████████ | 110/121 [02:24<00:13,  1.22s/it]
 92%|█████████▏| 111/121 [02:26<00:12,  1.22s/it]
 93%|█████████▎| 112/121 [02:27<00:11,  1.22s/it]
 93%|█████████▎| 113/121 [02:28<00:09,  1.22s/it]
 94%|█████████▍| 114/121 [02:29<00:08,  1.22s/it]
 95%|█████████▌| 115/121 [02:31<00:07,  1.22s/it]
 96%|█████████▌| 116/121 [02:32<00:06,  1.22s/it]
 97%|█████████▋| 117/121 [02:33<00:04,  1.22s/it]
 98%|█████████▊| 118/121 [02:34<00:03,  1.22s/it]
 98%|█████████▊| 119/121 [02:35<00:02,  1.23s/it]
 99%|█████████▉| 120/121 [02:37<00:01,  1.23s/it]
100%|██████████| 121/121 [02:38<00:00,  1.35s/it]
100%|██████████| 121/121 [02:38<00:00,  1.31s/it]
drift computed in  181.60s; total  181.61s

Extracting spikes using templates
Re-computing universal templates from data.

  0%|          | 0/121 [00:00<?, ?it/s]
  1%|          | 1/121 [00:01<02:27,  1.23s/it]
  2%|▏         | 2/121 [00:02<02:25,  1.22s/it]
  2%|▏         | 3/121 [00:03<02:23,  1.22s/it]
  3%|▎         | 4/121 [00:04<02:22,  1.22s/it]
  4%|▍         | 5/121 [00:06<02:20,  1.22s/it]
  5%|▍         | 6/121 [00:07<02:30,  1.31s/it]
  6%|▌         | 7/121 [00:08<02:25,  1.28s/it]
  7%|▋         | 8/121 [00:09<02:21,  1.25s/it]
  7%|▋         | 9/121 [00:11<02:18,  1.24s/it]
  8%|▊         | 10/121 [00:12<02:16,  1.23s/it]
  9%|▉         | 11/121 [00:13<02:14,  1.22s/it]
 10%|▉         | 12/121 [00:15<02:31,  1.39s/it]
 11%|█         | 13/121 [00:16<02:24,  1.33s/it]
 12%|█▏        | 14/121 [00:17<02:18,  1.30s/it]
 12%|█▏        | 15/121 [00:19<02:14,  1.27s/it]
 13%|█▎        | 16/121 [00:20<02:11,  1.25s/it]
 14%|█▍        | 17/121 [00:21<02:08,  1.24s/it]
 15%|█▍        | 18/121 [00:22<02:06,  1.23s/it]
 16%|█▌        | 19/121 [00:23<02:04,  1.22s/it]
 17%|█▋        | 20/121 [00:25<02:03,  1.22s/it]
 17%|█▋        | 21/121 [00:26<02:01,  1.22s/it]
 18%|█▊        | 22/121 [00:27<02:00,  1.21s/it]
 19%|█▉        | 23/121 [00:29<02:31,  1.54s/it]
 20%|█▉        | 24/121 [00:30<02:19,  1.44s/it]
 21%|██        | 25/121 [00:32<02:11,  1.37s/it]
 21%|██▏       | 26/121 [00:33<02:05,  1.32s/it]
 22%|██▏       | 27/121 [00:34<02:01,  1.29s/it]
 23%|██▎       | 28/121 [00:35<01:57,  1.26s/it]
 24%|██▍       | 29/121 [00:37<01:54,  1.25s/it]
 25%|██▍       | 30/121 [00:38<01:52,  1.24s/it]
 26%|██▌       | 31/121 [00:39<01:50,  1.23s/it]
 26%|██▋       | 32/121 [00:40<01:48,  1.22s/it]
 27%|██▋       | 33/121 [00:41<01:47,  1.22s/it]
 28%|██▊       | 34/121 [00:43<01:45,  1.22s/it]
 29%|██▉       | 35/121 [00:44<01:44,  1.21s/it]
 30%|██▉       | 36/121 [00:45<01:43,  1.21s/it]
 31%|███       | 37/121 [00:46<01:41,  1.21s/it]
 31%|███▏      | 38/121 [00:47<01:40,  1.21s/it]
 32%|███▏      | 39/121 [00:49<01:39,  1.21s/it]
 33%|███▎      | 40/121 [00:50<01:37,  1.21s/it]
 34%|███▍      | 41/121 [00:51<01:36,  1.21s/it]
 35%|███▍      | 42/121 [00:52<01:35,  1.21s/it]
 36%|███▌      | 43/121 [00:53<01:34,  1.21s/it]
 36%|███▋      | 44/121 [00:57<02:24,  1.87s/it]
 37%|███▋      | 45/121 [00:58<02:07,  1.67s/it]
 38%|███▊      | 46/121 [00:59<01:55,  1.53s/it]
 39%|███▉      | 47/121 [01:00<01:46,  1.44s/it]
 40%|███▉      | 48/121 [01:02<01:39,  1.37s/it]
 40%|████      | 49/121 [01:03<01:35,  1.32s/it]
 41%|████▏     | 50/121 [01:04<01:31,  1.29s/it]
 42%|████▏     | 51/121 [01:05<01:28,  1.26s/it]
 43%|████▎     | 52/121 [01:07<01:26,  1.25s/it]
 44%|████▍     | 53/121 [01:08<01:24,  1.24s/it]
 45%|████▍     | 54/121 [01:09<01:22,  1.23s/it]
 45%|████▌     | 55/121 [01:10<01:20,  1.22s/it]
 46%|████▋     | 56/121 [01:11<01:19,  1.22s/it]
 47%|████▋     | 57/121 [01:13<01:17,  1.21s/it]
 48%|████▊     | 58/121 [01:14<01:16,  1.21s/it]
 49%|████▉     | 59/121 [01:15<01:15,  1.21s/it]
 50%|████▉     | 60/121 [01:16<01:13,  1.21s/it]
 50%|█████     | 61/121 [01:17<01:12,  1.21s/it]
 51%|█████     | 62/121 [01:19<01:11,  1.21s/it]
 52%|█████▏    | 63/121 [01:20<01:10,  1.21s/it]
 53%|█████▎    | 64/121 [01:21<01:08,  1.21s/it]
 54%|█████▎    | 65/121 [01:22<01:07,  1.21s/it]
 55%|█████▍    | 66/121 [01:23<01:06,  1.21s/it]
 55%|█████▌    | 67/121 [01:25<01:05,  1.21s/it]
 56%|█████▌    | 68/121 [01:26<01:04,  1.21s/it]
 57%|█████▋    | 69/121 [01:27<01:02,  1.21s/it]
 58%|█████▊    | 70/121 [01:28<01:01,  1.21s/it]
 59%|█████▊    | 71/121 [01:29<01:00,  1.21s/it]
 60%|█████▉    | 72/121 [01:31<00:59,  1.21s/it]
 60%|██████    | 73/121 [01:32<00:58,  1.21s/it]
 61%|██████    | 74/121 [01:33<00:56,  1.21s/it]
 62%|██████▏   | 75/121 [01:34<00:55,  1.21s/it]
 63%|██████▎   | 76/121 [01:36<00:54,  1.21s/it]
 64%|██████▎   | 77/121 [01:37<00:53,  1.21s/it]
 64%|██████▍   | 78/121 [01:38<00:51,  1.21s/it]
 65%|██████▌   | 79/121 [01:39<00:50,  1.21s/it]
 66%|██████▌   | 80/121 [01:40<00:49,  1.21s/it]
 67%|██████▋   | 81/121 [01:42<00:48,  1.21s/it]
 68%|██████▊   | 82/121 [01:43<00:47,  1.21s/it]
 69%|██████▊   | 83/121 [01:44<00:45,  1.21s/it]
 69%|██████▉   | 84/121 [01:45<00:44,  1.21s/it]
 70%|███████   | 85/121 [01:46<00:43,  1.21s/it]
 71%|███████   | 86/121 [01:48<00:42,  1.21s/it]
 72%|███████▏  | 87/121 [01:49<00:41,  1.21s/it]
 73%|███████▎  | 88/121 [01:54<01:23,  2.54s/it]
 74%|███████▎  | 89/121 [01:56<01:08,  2.14s/it]
 74%|███████▍  | 90/121 [01:57<00:57,  1.86s/it]
 75%|███████▌  | 91/121 [01:58<00:49,  1.67s/it]
 76%|███████▌  | 92/121 [01:59<00:44,  1.53s/it]
 77%|███████▋  | 93/121 [02:01<00:40,  1.43s/it]
 78%|███████▊  | 94/121 [02:02<00:36,  1.37s/it]
 79%|███████▊  | 95/121 [02:03<00:34,  1.32s/it]
 79%|███████▉  | 96/121 [02:04<00:32,  1.28s/it]
 80%|████████  | 97/121 [02:05<00:30,  1.26s/it]
 81%|████████  | 98/121 [02:07<00:28,  1.25s/it]
 82%|████████▏ | 99/121 [02:08<00:27,  1.23s/it]
 83%|████████▎ | 100/121 [02:09<00:25,  1.23s/it]
 83%|████████▎ | 101/121 [02:10<00:24,  1.22s/it]
 84%|████████▍ | 102/121 [02:11<00:23,  1.22s/it]
 85%|████████▌ | 103/121 [02:13<00:21,  1.22s/it]
 86%|████████▌ | 104/121 [02:14<00:20,  1.21s/it]
 87%|████████▋ | 105/121 [02:15<00:19,  1.21s/it]
 88%|████████▊ | 106/121 [02:16<00:18,  1.21s/it]
 88%|████████▊ | 107/121 [02:17<00:16,  1.21s/it]
 89%|████████▉ | 108/121 [02:19<00:15,  1.21s/it]
 90%|█████████ | 109/121 [02:20<00:14,  1.21s/it]
 91%|█████████ | 110/121 [02:21<00:13,  1.21s/it]
 92%|█████████▏| 111/121 [02:22<00:12,  1.21s/it]
 93%|█████████▎| 112/121 [02:23<00:10,  1.21s/it]
 93%|█████████▎| 113/121 [02:25<00:09,  1.21s/it]
 94%|█████████▍| 114/121 [02:26<00:08,  1.21s/it]
 95%|█████████▌| 115/121 [02:27<00:07,  1.21s/it]
 96%|█████████▌| 116/121 [02:28<00:06,  1.21s/it]
 97%|█████████▋| 117/121 [02:30<00:04,  1.21s/it]
 98%|█████████▊| 118/121 [02:31<00:03,  1.21s/it]
 98%|█████████▊| 119/121 [02:32<00:02,  1.21s/it]
 99%|█████████▉| 120/121 [02:33<00:01,  1.21s/it]
100%|██████████| 121/121 [02:35<00:00,  1.31s/it]
100%|██████████| 121/121 [02:35<00:00,  1.28s/it]
22495125 spikes extracted in  156.14s; total  337.75s

First clustering

  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:15<01:01, 15.33s/it]
 40%|████      | 2/5 [00:22<00:32, 10.81s/it]
 60%|██████    | 3/5 [00:29<00:17,  8.71s/it]
 80%|████████  | 4/5 [00:33<00:07,  7.11s/it]
100%|██████████| 5/5 [00:35<00:00,  5.11s/it]
100%|██████████| 5/5 [00:35<00:00,  7.08s/it]
10 clusters found, in  36.13s; total  373.88s

Extracting spikes using cluster waveforms

  0%|          | 0/121 [00:00<?, ?it/s]
  0%|          | 0/121 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/cluster/home/sachur/Overlap_project/code/Python/Kilosort/kilosort4_multfile.py", line 193, in <module>
    sorting = ss.run_sorter_by_property('kilosort4',
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/launcher.py", line 297, in run_sorter_by_property
    sorting_list = run_sorter_jobs(job_list, engine=engine, engine_kwargs=engine_kwargs, return_output=True)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/launcher.py", line 106, in run_sorter_jobs
    sorting = run_sorter(**kwargs)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/runsorter.py", line 175, in run_sorter
    return run_sorter_local(**common_kwargs)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/runsorter.py", line 225, in run_sorter_local
    SorterClass.run_from_folder(output_folder, raise_error, verbose)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/basesorter.py", line 293, in run_from_folder
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/basesorter.py", line 258, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/spikeinterface/sorters/external/kilosort4.py", line 260, in _run_from_folder
    st, tF, _, _ = detect_spikes(ops, device, bfile, tic0=tic0, progress_bar=progress_bar)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/run_kilosort.py", line 409, in detect_spikes
    st, tF, ops = template_matching.extract(ops, bfile, Wall3, device=device,
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/template_matching.py", line 36, in extract
    stt, amps, Xres = run_matching(ops, X, U, ctc, device=device)
  File "/cluster/home/sachur/.local/lib/python3.10/site-packages/kilosort/template_matching.py", line 155, in run_matching
    st[k:k+nsp, 0] = iX[:,0]
RuntimeError: The expanded size of the tensor (1368) must match the existing size (2255) at non-singleton dimension 0.  Target sizes: [1368].  Tensor sizes: [2255]

Spike sorting failed. You can inspect the runtime trace in /cluster/home/sachur/Overlap_project/data/OpenEphys/65588_2024-03-08_16-10-29_C_ks4/0/spikeinterface_log.json.
zm711 commented 7 months ago

So that is still a KS4 error that we can't fix. So the last thing to test is can you try to run with all defaults (so we will do the CAR, the whitening, and the drift correction) and see if that works? KS does expect the data to have a CAR + whitening (drift correction is optional). So it might be that by skipping those steps the data structure is not quite what KS wants.

alejoe91 commented 7 months ago

Yeah Inhad troubles running KS4 without whitening! So whitening is definitely required!

Sachuriga commented 7 months ago

OK! that's good to know, let me try with whiten

Sachuriga commented 7 months ago

OK, I'm gonna close this issue. The above error was happens in the cluster PC, once I run the code in Local PC the code is working perfectly no matter I'm 'do_CAR':False or 'skip_kilosort_preprocessing':True. I the local PC I created the env that recommended by kilosort4 which is python==3.9 and cuda==11.8. However, with the cluster PC, I don't have as many options. The cluster environment uses python==3.10 and cuda==11.7. So, my guess is that some conflicts occur on the cluster PC when running Kilosort4. Later, I will try to run Kilosort4 with Singularity. That should solve the problem.

34 clusters found, in  4.57s; total  28.44s

Merging clusters
32 units found, in  0.10s; total  28.55s

Saving to phy and computing refractory periods
11 units found with good refractory periods

Total runtime: 28.65s = 00:00:28 h:m:s
========================================
Loading recording with SpikeInterface...
number of samples: 38698575
number of channels: 10
numbef of segments: 1
sampling rate: 30000.0
dtype: int16
========================================
Preprocessing filters computed in  0.11s; total  0.11s
Skipping drift correction.

Extracting spikes using templates
Re-computing universal templates from data.