Fix for plugins

[zm711]

@AceTylerCholine could you test this?

If this doesn't work I'll need to do more extensive testing (on Friday) to get this working.

[AceTylerCholine]

@zm711

This didn't work, sorry. The easiest workaround for me was to just revert back to imp since I can't get the rest of Phy to work with 3.12 anyways, but I had created another workaround that tries to use importlib and reverts back to imp if importlib didn't work here: https://github.com/AceTylerCholine/phy/tree/plugin-imp

This is what I got when I ran it with this commit: (phy2) PS C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy> phy template-gui params.py 22:37:53.381 [W] model:625 Unreferenced clusters found in spike_clusters (generally not a problem) 22:37:53.443 [W] model:667 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed. 22:37:55.170 [W] plugin:153 The plugin ISIViolationPlugin1ms couldn't be found. 22:37:55.170 [W] plugin:153 The plugin ISIViolationPlugin1_5ms couldn't be found. 22:37:55.170 [W] plugin:153 The plugin ISIViolationPlugin2ms couldn't be found.

It wasn't a fatal error, but the plugins didn't load. With the workaround from my branch I get this: (phy2) PS C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy> phy template-gui params.py 22:40:21.713 [W] model:625 Unreferenced clusters found in spike_clusters (generally not a problem) 22:40:21.765 [W] model:667 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed. 22:40:22.734 [W] plugin:112 Could not find module: 1ms_isi_violations. Trying imp.find_module as fallback. 22:40:22.734 [W] plugin:112 Could not find module: 1_5ms_isi_violations. Trying imp.find_module as fallback. 22:40:22.749 [W] plugin:112 Could not find module: 2ms_isi_violations. Trying imp.find_module as fallback.

and the plugins do load, but it doesn't work with 3.12 either way for me, so I'm not sure if my branch is actually useful at solving this problem.

[zm711]

Actually that's an improvement. I think I need to tweak how the folder is located for these plugins. So I'll try to keep working on this a bit. I think we are close. I'll ping you when I've add some changes!

[zm711]

@AceTylerCholine could you try from this PR now?

My hope is to get CI working on the repo soon to prevent this in the future :)

[AceTylerCholine]

@zm711 no, sorry, still couldn't load the plugins. It didn't cause a fatal error though, just didn't load them.

(phy2) PS C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy> phy template-gui params.py 08:28:42.949 [W] model:625 Unreferenced clusters found in spike_clusters (generally not a problem) 08:28:43.567 [W] model:667 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed. 08:28:45.866 [W] plugin:153 The plugin ISIViolationPlugin1ms couldn't be found. 08:28:45.866 [W] plugin:153 The plugin ISIViolationPlugin1_5ms couldn't be found. 08:28:45.866 [W] plugin:153 The plugin ISIViolationPlugin2ms couldn't be found.

[zm711]

Sorry for keep making you try, but could you try the latest push? @AceTylerCholine

[AceTylerCholine]

No worries, happy to help @zm711 !

Unfortunately, same thing with that push:

(phy2) PS C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy> phy template-gui params.py 11:06:49.364 [W] model:625 Unreferenced clusters found in spike_clusters (generally not a problem) 11:06:49.474 [W] model:667 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed. 11:06:50.792 [W] plugin:153 The plugin ISIViolationPlugin1ms couldn't be found. 11:06:50.807 [W] plugin:153 The plugin ISIViolationPlugin1_5ms couldn't be found. 11:06:50.807 [W] plugin:153 The plugin ISIViolationPlugin2ms couldn't be found.

[zm711]

Could you rerun

phy template-gui params.py --debug

So we can see all logging info?

[AceTylerCholine]

@zm711 of course, sorry about that:

(phy2) PS C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy> phy template-gui params.py --debug
12:19:54.227 [D] __init__:68          Start capturing exceptions.
12:19:54.384 [D] model:619            Loading spike clusters.
12:19:54.431 [W] model:625            Unreferenced clusters found in spike_clusters (generally not a problem)
12:19:54.478 [D] model:569            No channel shank file found.
12:19:54.478 [D] model:692            Loading templates.
12:19:54.525 [D] model:720            Templates are sparse.
12:19:54.525 [W] model:667            Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed.
12:19:54.525 [D] model:730            Loading the whitening matrix.
12:19:54.525 [D] model:434            Whitening matrix file not found.
12:19:54.525 [D] model:737            Loading the inverse of the whitening matrix.
12:19:54.573 [D] model:766            Loading features.
12:19:54.598 [D] model:781            Features are sparse.
12:19:54.598 [D] model:803            Loading template features.
12:19:54.600 [D] model:504            Load `cluster_channel_group.tsv`.
12:19:54.620 [D] model:504            Load `cluster_group.tsv`.
12:19:54.622 [D] model:504            Load `cluster_si_unit_id.tsv`.
12:19:54.637 [D] model:504            Load `cluster_si_unit_ids.tsv`.
12:19:55.698 [D] context:100          Initialize joblib cache dir at `C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy\.phy`.
12:19:55.698 [D] context:101          Reducing the size of the cache if needed.
12:19:55.918 [D] base:102             Add filter `high_pass`.
12:19:55.918 [D] config:31            Load config file `C:\Users\short\.phy\phy_config.py`.
12:19:55.934 [D] plugin:75            Scanning `C:\Users\short\.phy\plugins`.
12:19:55.949 [D] plugin:79            Found plugin module `1ms_isi_violations.py`.
12:19:55.949 [D] plugin:79            Found plugin module `1_5ms_isi_violations.py`.
12:19:55.949 [D] plugin:79            Found plugin module `2ms_isi_violations.py`.
12:19:55.949 [D] plugin:147           Loading 3 plugins.
12:19:55.949 [W] plugin:153           The plugin ISIViolationPlugin1ms couldn't be found.
12:19:55.949 [W] plugin:153           The plugin ISIViolationPlugin1_5ms couldn't be found.
12:19:55.949 [W] plugin:153           The plugin ISIViolationPlugin2ms couldn't be found.
12:19:55.949 [D] context:126          Load memcache for `phy.apps.base._get_mean_waveforms`.
12:19:55.969 [D] context:126          Load memcache for `phy.apps.base._get_mean_waveforms`.
12:19:55.969 [D] context:126          Load memcache for `phy.apps.base._get_template_waveforms`.
12:19:55.981 [D] context:126          Load memcache for `phy.apps.base.get_mean_spike_template_amplitudes`.
12:19:55.981 [D] context:126          Load memcache for `phy.apps.base.get_template_counts`.
12:19:55.997 [D] context:126          Load memcache for `phy.apps.base.get_template_for_cluster`.
12:19:55.997 [D] context:126          Load memcache for `phy.apps.template.gui.get_template_amplitude`.
12:19:56.012 [D] context:126          Load memcache for `phy.apps.base.get_cluster_amplitude`.
12:19:56.012 [D] context:126          Load memcache for `phy.apps.base.get_mean_firing_rate`.
12:19:56.012 [D] context:126          Load memcache for `phy.apps.base.get_best_channel`.
12:19:56.028 [D] context:126          Load memcache for `phy.apps.template.gui.get_best_channels`.
12:19:56.043 [D] context:126          Load memcache for `phy.apps.base.get_channel_shank`.
12:19:56.043 [D] context:126          Load memcache for `phy.apps.base.get_probe_depth`.
12:19:56.059 [D] context:126          Load memcache for `phy.apps.base.peak_channel_similarity`.
12:19:56.106 [D] context:185          Save data to `C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy\.phy\spikes_per_cluster.pkl`.
12:19:56.122 [D] gui:463              Creating GUI.
12:19:56.201 [D] state:46             Load C:\Users\short\.phy\TemplateGUI\state.json for GUIState.
12:19:56.216 [D] state:46             Load C:\Users\short\Documents\spike_sorting\20240323_144517_alone_comp_subj_3-1_t5b5_merged.rec\phy\.phy\state.json for GUIState.
12:19:57.601 [D] gui:718              Add view ClusterView to GUI.
12:19:57.675 [D] gui:718              Add view SimilarityView to GUI.
12:19:57.780 [D] gui:718              Add view WaveformView to GUI.
12:19:58.114 [D] base:337             Set state for WaveformView.
12:19:58.145 [D] gui:718              Add view CorrelogramView to GUI.
12:19:58.148 [D] base:337             Set state for CorrelogramView.
12:19:58.195 [D] gui:718              Add view FeatureView to GUI.
12:19:58.198 [D] base:337             Set state for FeatureView.
12:19:58.254 [D] gui:718              Add view AmplitudeView to GUI.
12:19:58.259 [D] base:337             Set state for AmplitudeView.
12:19:58.734 [D] gui:519              Load the geometry state.
12:20:01.160 [D] supervisor:817       Clusters selected: [3] (4)

So it's able to load the .py files as plugins, reads the class within the scripts, but can't load them? You don't need me to analyze it for you though.

[zm711]

@AceTylerCholine,

So I set this up myself locally and it seems to be working for me....

Not sure what the problem is? Could you make sure you have the most recent commit?

[zm711]

After this fix you see the util tests are passing

https://github.com/zm711/phy/actions/runs/9390953253

I'm still fighting with the other tests. I'm going to merge this now along with a general limit to 3.11 because there are some issues with OpenGL and 3.12.

[AceTylerCholine]

Reinstalled with the most recent push and my plugins work again! Thanks so much!

[zm711]

Of course @AceTylerCholine Happy it worked!