Closed Tampu closed 6 months ago
marius10p
commented
3 years ago If there is really only one cluster, there are probably many squeezes along the way both in Kilosort and Phy not designed to deal with singleton dimensions. I recommend to lower the default thresholds (ops.Th) for this particular recording so you get some MUA in there, and then it should no longer fail.
Tampu
commented
3 years ago Thank you, I tried many combinations and ops.Th = [7 8] seems to work for this recording.
Just a small issue, feature view is not getting created and trace view doesn't load completely. I have hdf5 recordings and I tried saving it as .bin directly and also converting into .bin using the hdf5 dump utility. In both the cases, I am not able to get the trace view.
marius10p
commented
3 years ago The feature view is not yet enabled in Kilosort3. Not sure about the trace view, what do you mean it doesn't load completely? What errors do you get in the command line in Phy?
Tampu
commented
3 years ago Ah, okay. Sorry, I wasn't clear. I meant when I select a cluster, the events from this cluster are not highlighted. I do not get any error in the command line as you can see below but the trace view looks like this:
phy template-gui params.py --debug [90m12:10:29.135 [D] init:68 Start capturing exceptions.[0m [90m12:10:29.509 [D] model:560 Loading spike clusters.[0m [90m12:10:29.512 [D] model:507 No channel shank file found.[0m [90m12:10:29.513 [D] model:627 Loading templates.[0m [90m12:10:29.515 [D] model:654 Templates are dense.[0m [90m12:10:29.515 [D] model:604 Loading spike waveforms.[0m [90m12:10:29.515 [D] model:609 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed.[0m [90m12:10:29.515 [D] model:660 Loading the whitening matrix.[0m [90m12:10:29.517 [D] model:667 Loading the inverse of the whitening matrix.[0m [90m12:10:29.518 [D] model:164 Loading traces at
D:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\temp_wh.dat.[0m [90m12:10:29.574 [D] model:696 Loading features.[0m [90m12:10:29.575 [D] model:733 Loading template features.[0m [90m12:10:29.577 [D] model:442 Loadcluster_Amplitude.tsv.[0m [90m12:10:29.577 [D] _misc:319 Read D:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2\cluster_Amplitude.tsv.[0m [90m12:10:29.578 [D] model:442 Loadcluster_ContamPct.tsv.[0m [90m12:10:29.578 [D] _misc:319 Read D:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2\cluster_ContamPct.tsv.[0m [90m12:10:29.578 [D] model:442 Loadcluster_group.tsv.[0m [90m12:10:29.579 [D] _misc:319 Read D:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2\cluster_group.tsv.[0m [90m12:10:29.579 [D] model:442 Loadcluster_KSLabel.tsv.[0m [90m12:10:29.580 [D] _misc:319 Read D:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2\cluster_KSLabel.tsv.[0m [90m12:10:29.609 [D] context:100 Initialize joblib cache dir atD:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2\.phy.[0m [90m12:10:29.609 [D] context:101 Reducing the size of the cache if needed.[0m [90m12:10:29.637 [D] base:102 Add filterhigh_pass.[0m [90m12:10:29.638 [D] config:31 Load config fileC:\Users\Lab Pc\.phy\phy_config.py.[0m [90m12:10:29.639 [D] plugin:145 Loading 0 plugins.[0m [90m12:10:29.639 [D] context:126 Load memcache forphy.apps.base._get_mean_waveforms.[0m [90m12:10:29.640 [D] context:126 Load memcache forphy.apps.base._get_mean_waveforms.[0m [90m12:10:29.641 [D] context:126 Load memcache forphy.apps.base._get_template_waveforms.[0m [90m12:10:29.647 [D] context:126 Load memcache forphy.apps.base.get_mean_spike_template_amplitudes.[0m [90m12:10:29.649 [D] context:126 Load memcache forphy.apps.base.get_template_counts.[0m [90m12:10:29.650 [D] context:126 Load memcache forphy.apps.base.get_template_for_cluster.[0m [90m12:10:29.650 [D] context:126 Load memcache forphy.apps.template.gui.get_template_amplitude.[0m [90m12:10:29.651 [D] context:126 Load memcache forphy.apps.base.get_cluster_amplitude.[0m [90m12:10:29.652 [D] context:126 Load memcache forphy.apps.base.get_mean_firing_rate.[0m [90m12:10:29.653 [D] context:126 Load memcache forphy.apps.base.get_best_channel.[0m [90m12:10:29.653 [D] context:126 Load memcache forphy.apps.template.gui.get_best_channels.[0m [90m12:10:29.654 [D] context:126 Load memcache forphy.apps.base.get_channel_shank.[0m [90m12:10:29.655 [D] context:126 Load memcache forphy.apps.base.get_probe_depth.[0m [90m12:10:29.656 [D] context:126 Load memcache forphy.apps.base.peak_channel_similarity.[0m [90m12:10:29.658 [D] context:209 The fileD:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2\.phy\new_cluster_id.pkldoesn't exist.[0m [90m12:10:29.662 [D] context:185 Save data toD:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2\.phy\spikes_per_cluster.pkl.[0m [90m12:10:29.669 [D] gui:463 Creating GUI.[0m [90m12:10:29.671 [D] state:46 Load C:\Users\Lab Pc.phy\TemplateGUI\state.json for GUIState.[0m [90m12:10:29.677 [D] state:46 Load D:\raviku53\Kilosort3\Kilosort-main\Kilosort-main\mydata\bin_rec\kilosort3_5\2.phy\state.json for GUIState.[0m [90m12:10:30.016 [D] gui:718 Add view ClusterView to GUI.[0m [90m12:10:30.260 [D] gui:718 Add view SimilarityView to GUI.[0m [90m12:10:30.422 [D] gui:718 Add view WaveformView to GUI.[0m [90m12:10:30.426 [D] base:321 Set state for WaveformView.[0m [90m12:10:30.452 [D] gui:718 Add view WaveformView to GUI.[0m [90m12:10:30.455 [D] base:321 Set state for WaveformView (1).[0m [90m12:10:30.481 [D] gui:718 Add view WaveformView to GUI.[0m [90m12:10:30.484 [D] base:321 Set state for WaveformView (2).[0m [90m12:10:30.511 [D] gui:718 Add view WaveformView to GUI.[0m [90m12:10:30.514 [D] base:321 Set state for WaveformView (3).[0m [90m12:10:30.542 [D] gui:718 Add view WaveformView to GUI.[0m [90m12:10:30.544 [D] base:321 Set state for WaveformView (4).[0m [90m12:10:30.572 [D] gui:718 Add view WaveformView to GUI.[0m [90m12:10:30.575 [D] base:321 Set state for WaveformView (5).[0m [90m12:10:30.592 [D] gui:718 Add view CorrelogramView to GUI.[0m [90m12:10:30.594 [D] base:321 Set state for CorrelogramView.[0m [90m12:10:30.609 [D] gui:718 Add view CorrelogramView to GUI.[0m [90m12:10:30.611 [D] base:321 Set state for CorrelogramView (1).[0m [90m12:10:30.643 [D] gui:718 Add view AmplitudeView to GUI.[0m [90m12:10:30.646 [D] base:321 Set state for AmplitudeView.[0m [90m12:10:30.693 [D] trace:367 Redraw the entire trace view.[0m [90m12:10:30.855 [D] trace:332 Redraw the entire trace view.[0m [90m12:10:31.317 [D] gui:718 Add view TraceView to GUI.[0m [90m12:10:31.319 [D] base:321 Set state for TraceView.[0m [90m12:10:31.319 [D] base:477 Set color scheme to random.[0m [90m12:10:31.668 [D] base:1564 Adding default color schemes to TraceView.[0m [90m12:10:31.870 [D] gui:718 Add view TemplateView to GUI.[0m [90m12:10:31.873 [D] base:321 Set state for TemplateView.[0m [90m12:10:31.873 [D] base:477 Set color scheme to random.[0m [90m12:10:31.874 [D] base:1564 Adding default color schemes to TemplateView.[0m [90m12:10:31.902 [D] gui:718 Add view TemplateView to GUI.[0m [90m12:10:31.905 [D] base:321 Set state for TemplateView (1).[0m [90m12:10:31.905 [D] base:477 Set color scheme to random.[0m [90m12:10:31.906 [D] base:1564 Adding default color schemes to TemplateView (1).[0m [90m12:10:31.935 [D] gui:718 Add view TemplateView to GUI.[0m [90m12:10:31.938 [D] base:321 Set state for TemplateView (2).[0m [90m12:10:31.938 [D] base:477 Set color scheme to random.[0m [90m12:10:31.939 [D] base:1564 Adding default color schemes to TemplateView (2).[0m [90m12:10:31.942 [D] gui:519 Load the geometry state.[0m [90m12:10:32.499 [D] supervisor:812 Clusters selected: [0] (8)[0m [90m12:11:16.443 [D] trace:367 Redraw the entire trace view.[0m [90m12:11:16.607 [D] trace:332 Redraw the entire trace view.[0m [90m12:11:17.077 [D] gui:718 Add view TraceView to GUI.[0m [90m12:11:17.347 [D] base:321 Set state for TraceView (1).[0m [90m12:11:17.347 [D] base:477 Set color scheme to random.[0m [90m12:11:17.693 [D] base:1564 Adding default color schemes to TraceView (1).[0m [90m12:11:23.297 [D] trace:367 Redraw the entire trace view.[0m [90m12:11:23.298 [D] trace:705 Redraw the entire trace view.[0m [90m12:11:23.803 [D] gui:718 Add view TraceImageView to GUI.[0m [90m12:11:23.811 [D] base:321 Set state for TraceImageView.[0m [90m12:11:23.812 [D] base:477 Set color scheme to random.[0m [90m12:11:23.812 [D] trace:705 Redraw the entire trace view.[0m [90m12:11:24.303 [D] base:1564 Adding default color schemes to TraceImageView.[0m [90m12:11:32.424 [D] base:261 Close view WaveformView.[0m [90m12:11:32.425 [D] state:171 Update GUI state for WaveformView[0m [90m12:11:32.427 [D] base:261 Close view WaveformView (1).[0m [90m12:11:32.427 [D] state:171 Update GUI state for WaveformView (1)[0m [90m12:11:32.427 [D] base:261 Close view WaveformView (2).[0m [90m12:11:32.428 [D] state:171 Update GUI state for WaveformView (2)[0m [90m12:11:32.428 [D] base:261 Close view WaveformView (3).[0m [90m12:11:32.429 [D] state:171 Update GUI state for WaveformView (3)[0m [90m12:11:32.429 [D] base:261 Close view WaveformView (4).[0m [90m12:11:32.429 [D] state:171 Update GUI state for WaveformView (4)[0m [90m12:11:32.430 [D] base:261 Close view WaveformView (5).[0m [90m12:11:32.430 [D] state:171 Update GUI state for WaveformView (5)[0m [90m12:11:32.431 [D] base:261 Close view CorrelogramView.[0m [90m12:11:32.431 [D] state:171 Update GUI state for CorrelogramView[0m [90m12:11:32.431 [D] base:261 Close view CorrelogramView (1).[0m [90m12:11:32.432 [D] state:171 Update GUI state for CorrelogramView (1)[0m [90m12:11:32.432 [D] base:261 Close view AmplitudeView.[0m [90m12:11:32.433 [D] state:171 Update GUI state for AmplitudeView[0m [90m12:11:32.433 [D] base:261 Close view TraceView.[0m [90m12:11:32.433 [D] state:171 Update GUI state for TraceView[0m [90m12:11:32.434 [D] base:261 Close view TemplateView.[0m [90m12:11:32.434 [D] state:171 Update GUI state for TemplateView[0m [90m12:11:32.435 [D] base:261 Close view TemplateView (1).[0m [90m12:11:32.439 [D] state:171 Update GUI state for TemplateView (1)[0m [90m12:11:32.439 [D] base:261 Close view TemplateView (2).[0m [90m12:11:32.440 [D] state:171 Update GUI state for TemplateView (2)[0m [90m12:11:32.440 [D] base:261 Close view TraceView (1).[0m [90m12:11:32.441 [D] state:171 Update GUI state for TraceView (1)[0m [90m12:11:32.441 [D] base:261 Close view TraceImageView.[0m [90m12:11:32.441 [D] state:171 Update GUI state for TraceImageView[0m [90m12:11:36.301 [D] gui:718 Add view RasterView to GUI.[0m [90m12:11:36.310 [D] base:321 Set state for RasterView.[0m [90m12:11:36.311 [D] base:477 Set color scheme to random.[0m [90m12:11:36.314 [D] base:1564 Adding default color schemes to RasterView.[0m [90m12:11:44.738 [D] base:261 Close view WaveformView.[0m [90m12:11:44.739 [D] state:171 Update GUI state for WaveformView[0m [90m12:11:44.741 [D] base:261 Close view WaveformView (1).[0m [90m12:11:44.741 [D] state:171 Update GUI state for WaveformView (1)[0m [90m12:11:44.742 [D] base:261 Close view WaveformView (2).[0m [90m12:11:44.742 [D] state:171 Update GUI state for WaveformView (2)[0m [90m12:11:44.742 [D] base:261 Close view WaveformView (3).[0m [90m12:11:44.743 [D] state:171 Update GUI state for WaveformView (3)[0m [90m12:11:44.743 [D] base:261 Close view WaveformView (4).[0m [90m12:11:44.744 [D] state:171 Update GUI state for WaveformView (4)[0m [90m12:11:44.744 [D] base:261 Close view WaveformView (5).[0m [90m12:11:44.745 [D] state:171 Update GUI state for WaveformView (5)[0m [90m12:11:44.745 [D] base:261 Close view CorrelogramView.[0m [90m12:11:44.745 [D] state:171 Update GUI state for CorrelogramView[0m [90m12:11:44.746 [D] base:261 Close view CorrelogramView (1).[0m [90m12:11:44.746 [D] state:171 Update GUI state for CorrelogramView (1)[0m [90m12:11:44.747 [D] base:261 Close view AmplitudeView.[0m [90m12:11:44.747 [D] state:171 Update GUI state for AmplitudeView[0m [90m12:11:44.747 [D] base:261 Close view TraceView.[0m [90m12:11:44.748 [D] state:171 Update GUI state for TraceView[0m [90m12:11:44.748 [D] base:261 Close view TemplateView.[0m [90m12:11:44.749 [D] state:171 Update GUI state for TemplateView[0m [90m12:11:44.753 [D] base:261 Close view TemplateView (1).[0m [90m12:11:44.753 [D] state:171 Update GUI state for TemplateView (1)[0m [90m12:11:44.754 [D] base:261 Close view TemplateView (2).[0m [90m12:11:44.754 [D] state:171 Update GUI state for TemplateView (2)[0m [90m12:11:44.755 [D] base:261 Close view TraceView (1).[0m [90m12:11:44.755 [D] state:171 Update GUI state for TraceView (1)[0m [90m12:11:44.755 [D] base:261 Close view TraceImageView.[0m [90m12:11:44.756 [D] state:171 Update GUI state for TraceImageView[0m [90m12:11:44.757 [D] base:261 Close view RasterView.[0m [90m12:11:44.757 [D] state:171 Update GUI state for RasterView[0m [90m12:11:44.757 [D] base:261 Close view RasterView.[0m [90m12:11:44.757 [D] state:171 Update GUI state for RasterView[0m [90m12:16:48.149 [D] supervisor:812 Clusters selected: [0] (23)[0m [90m12:16:48.720 [D] supervisor:834 Similar clusters selected: [1] (26)[0m [90m12:16:49.782 [D] supervisor:834 Similar clusters selected: [26] (3)[0m [90m12:16:50.507 [D] supervisor:834 Similar clusters selected: [3] (4)[0m [90m12:16:51.098 [D] supervisor:834 Similar clusters selected: [4] (5)[0m [90m12:16:52.713 [D] supervisor:834 Similar clusters selected: [25] (14)[0m [90m12:16:53.672 [D] supervisor:812 Clusters selected: [0] (23)[0m [90m12:16:55.986 [D] base:296 Enable auto update for TraceView (1).[0m [90m12:16:56.949 [D] base:296 Disable auto update for TraceView (1).[0m [90m12:16:57.698 [D] base:296 Enable auto update for TraceView (1).[0m [90m12:16:59.951 [D] supervisor:812 Clusters selected: [0] (23)[0m
marius10p
commented
3 years ago I don't recognize anything familiar in that traceview... Can you zoom in? I see some oblique lines through it, those could be indicative of a channel number mismatch between the data and what you told Kilosort. The Kilosort GUI would be a good place to figure out the right number of channels.
Tampu
commented
3 years ago This is the zoomed in trace view:
Kilosort guesses it as 270 channels as we can see below but the channel_map.npy output file after sorting using the script has 256 channels which is correct according to the channel map.
I am converting the hdf5 file using hdf5 dump facility as mentioned previously, could this be the reason for not getting the trace view? Also, I tried saving it as a bin file directly without using the dump facility, though I am able to run kilosort, the recording doesn't get loaded in the GUI (meaning it says cannot guess the channel number and when I entire manually, it still doesn't recognize it) and can't view the highlighted clusters in trace view. Could you please suggest which is the best method to convert hdf5 files for Kilosort?
Thank you very much for your time.
warnerwarner
commented
3 years ago Hi, is there any update on advice on saving hdf5 files for Kilosort. I'm struggling to find the best solution.
Also Tampu, if you're still struggling with this, I realised that h5dump makes a binary file which is row ordered, and matlab reads it in as column ordered so all your values are going to be muddled. In addition, if your datatype isn't int16 Kilosort will read it in incorrectly.
Tampu
commented
3 years ago Hi Warner, yes indeed I realized that h5 dump utility cannot be used directly to convert raw hdf5 recordings to binary format. Right now I am loading the hdf5 file using h5read in MATLAB, doing int16 conversion and saving it as a binary file before running Kilosort and its working fine.
Hi,
I tried Kilosort3 on a recording whose output I was able to load into phy2 while using Kilosort2.5(where the clusters were only mua's) but I get the following assertion error while using Kilosort3(one good cluster found, no mua's).
[90m23:17:23.383 [D] init:68 Start capturing exceptions.[0m [90m23:17:23.765 [D] model:560 Loading spike clusters.[0m [90m23:17:23.769 [D] model:507 No channel shank file found.[0m [90m23:17:23.769 [D] model:627 Loading templates.[0m [90m23:17:23.771 [D] model:654 Templates are dense.[0m [90m23:17:23.771 [D] model:604 Loading spike waveforms.[0m [90m23:17:23.771 [D] model:609 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed.[0m [90m23:17:23.771 [D] model:660 Loading the whitening matrix.[0m [90m23:17:23.773 [D] model:667 Loading the inverse of the whitening matrix.[0m [31m23:17:23.777 [E] init:62 An error has occurred (AssertionError): Traceback (most recent call last): File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\ProgramData\Anaconda3\Scripts\phy.exe__main.py", line 7, in
sys.exit(phycli())
File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 764, in call
return self.main(args, kwargs)
File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 717, in main
rv = self.invoke(ctx)
File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 956, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 555, in invoke
return callback(args, *kwargs)
File "c:\programdata\anaconda3\lib\site-packages\click\decorators.py", line 17, in new_func
return f(get_current_context(), args, kwargs)
File "c:\programdata\anaconda3\lib\site-packages\phy\apps__init__.py", line 159, in cli_template_gui
template_gui(params_path, kwargs)
File "c:\programdata\anaconda3\lib\site-packages\phy\apps\template\gui.py", line 198, in template_gui
controller = TemplateController(model=load_model(params_path), dir_path=dir_path, kwargs)
File "c:\programdata\anaconda3\lib\site-packages\phylib\io\model.py", line 1178, in load_model
return TemplateModel(get_template_params(params_path))
File "c:\programdata\anaconda3\lib\site-packages\phylib\io\model.py", line 297, in init__
self._load_data()
File "c:\programdata\anaconda3\lib\site-packages\phylib\io\model.py", line 386, in _load_data
assert self.similar_templates.shape == (self.n_templates, self.n_templates)
AssertionError
[0m
I attach the contents of similar_templates file for your reference. readNPY('similar_templates.npy')
ans =
single
Kindly have a look at this, thank you for your time.