WaveformView won't show up

[Antho2422]

Hi !

I'm working on a project were I'm performing spike sorting using SpikeInterface (SpikingCircus2 as sorter) and I'm trying to use Phy on my data. I made it work fine a few weeks ago but now, my data is loading but the WaveformView panel don't show the traces.

I though this was coming from some environment updates, so I tried to reinstall phy from this environment : https://github.com/cortex-lab/phy/blob/master/environment.yml

Unfortunately it did not fix the issue.

This is the error I get :

  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\gui\qt.py", line 547, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\cluster\views\base.py", line 175, in _worker  
    self.on_select(cluster_ids=cluster_ids, **kwargs)
  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\cluster\views\base.py", line 143, in on_select
    self.plot(**kwargs)
  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\cluster\views\waveform.py", line 311, in plot 
    bunchs = self.get_clusters_data()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\cluster\views\waveform.py", line 189, in get_clusters_data
    bunchs = [
             ^
  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\cluster\views\waveform.py", line 190, in <listcomp>
    self.waveforms_types.get()(cluster_id) for cluster_id in self.cluster_ids]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\utils\context.py", line 154, in memcached
    out = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "C:\Users\user.name\Anaconda3\envs\phy2\Lib\site-packages\phy\apps\base.py", line 562, in _get_template_waveforms
    data[i][:, b.channel_ids] = b.template

And this is what the gui is showing up :

Any help would be appreciated 😃

Thanks, Anthony

[Antho2422]

I also get the following error when I click on a cluster :

C:\Users\user.name\AppData\Local\pypoetry\Cache\virtualenvs\priam-5spsvreu-py3.9\lib\site-packages\joblib\_store_backends.py:207: CacheWarning: Unable to cache to disk. Possibly a race condition in the creation of the directory. Exception: [Errno 2] No such file or directory: '\\\\l2export\\iss02.xxx\\analyses\\user.name\\git_lustre\\priam\\data\\processed\\model_name\\03231\\preprocessed\\phy\\1\\.phy\\phy\\apps\\base\\TemplateMixin\\get_spike_template_amplitudes\\3cb821db3a3a89ad88d836c6936f64ae\\output.pkl.thread-1796794117808-pid-11172'.
  warnings.warn(

[zm711]

@Antho2422

I think you cutoff the error in the trace. Could you post the full error traceback? I see the lines where the error occurred, but not the final error. It would be helpful to figure out if this is a spikeinterface problem or a phy problem.

[Antho2422]

Thanks for your answer @zm711 . Sorry for cutting the error I didn't notice!

Here is the last line :

AttributeError: 'NoneType' object has no attribute 'template'

I'm afraid that is the only line I forgot...

Here is also a preview of the content of the phy folder. The spike_templates.npy file is here.

There is also the log file in case you need it : phy.log

Cheers :)

[zm711]

@Antho2422

Could you run with --debug?

so : phy template-gui params.py --debug

So we can see all the debug info as well. I think I have an idea of what is going on. I may need you to share the templates.npy file after the debugging.

Best, Zach

[Antho2422]

Okay thank you I ran it in debug mode and this is the output :

17:21:27.619 [D] __init__:68          Start capturing exceptions.
17:21:27.733 [W] model:1408           File \network\lustre\iss02\charpier\analyses\anthony.pinto\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\recording.dat does not exist.
17:21:28.296 [D] model:619            Loading spike clusters.
17:21:28.473 [D] model:569            No channel shank file found.
17:21:28.474 [D] model:692            Loading templates.
17:21:28.498 [W] model:667            Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed.
17:21:28.499 [D] model:730            Loading the whitening matrix.
17:21:28.500 [D] model:434            Whitening matrix file not found.
17:21:28.500 [D] model:737            Loading the inverse of the whitening matrix.
17:21:28.501 [D] model:440            Whitening matrix inverse file not found, computing it.
17:21:28.501 [D] model:744            Inversing the whitening matrix (4, 4).
17:21:28.669 [W] traces:462           File \\l2export\iss02.charpier\network\lustre\iss02\charpier\analyses\anthony.pinto\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\recording.dat does not exist.
17:21:28.670 [W] traces:462           File \\l2export\iss02.charpier\network\lustre\iss02\charpier\analyses\anthony.pinto\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\recording.dat does not exist.
17:21:28.671 [D] model:766            Loading features.
17:21:28.953 [D] model:781            Features are sparse.
17:21:28.959 [D] model:803            Loading template features.
17:21:28.981 [D] model:504            Load `cluster_amplitude_cutoff.tsv`.
17:21:29.002 [D] model:504            Load `cluster_amplitude_median.tsv`.
17:21:29.026 [D] model:504            Load `cluster_channel_group.tsv`.
17:21:29.074 [D] model:504            Load `cluster_group.tsv`.
17:21:29.089 [D] model:504            Load `cluster_isi_violations_count.tsv`.
17:21:29.118 [D] model:504            Load `cluster_isi_violations_ratio.tsv`.
17:21:29.134 [D] model:504            Load `cluster_presence_ratio.tsv`.
17:21:29.150 [D] model:504            Load `cluster_rp_contamination.tsv`.
17:21:29.164 [D] model:504            Load `cluster_rp_violations.tsv`.
17:21:29.183 [D] model:504            Load `cluster_si_unit_ids.tsv`.
17:21:29.200 [D] model:504            Load `cluster_sliding_rp_violation.tsv`.
17:21:29.237 [D] model:504            Load `cluster_snr.tsv`.
17:21:29.659 [D] context:80           Create cache directory `\\l2export\iss02.charpier\analyses\anthony.pinto\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\.phy`.
17:21:29.968 [D] context:100          Initialize joblib cache dir at `\\l2export\iss02.charpier\analyses\anthony.pinto\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\.phy`.
17:21:29.987 [D] context:101          Reducing the size of the cache if needed.
17:21:30.093 [D] base:102             Add filter `high_pass`.
17:21:30.125 [D] config:31            Load config file `C:\Users\anthony.pinto\.phy\phy_config.py`.
17:21:30.162 [D] plugin:145           Loading 0 plugins.
17:21:31.068 [D] context:209          The file `\\l2export\iss02.charpier\analyses\user.name\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\.phy\new_cluster_id.pkl` doesn't exist. 
17:21:31.097 [D] context:209          The file `\\l2export\iss02.charpier\analyses\anthony.pinto\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\.phy\spikes_per_cluster.pkl` doesn't exist.
17:21:31.119 [D] clustering:237       Recompute spikes_per_cluster manually: this might take a while.
17:21:31.269 [D] context:185          Save data to `\\l2export\iss02.charpier\analyses\anthony.pinto\git_lustre\priam\data\processed\golgi_v1.0\02711_2019-04-18_11-04\preprocessed\phy\1\.phy\spikes_per_cluster.pkl`.       
17:21:31.562 [D] gui:463              Creating GUI.
17:21:31.862 [D] state:46             Load C:\Users\anthony.pinto\.phy\TemplateGUI\state.json for GUIState.
17:21:32.592 [D] gui:718              Add view ClusterView to GUI.
17:21:32.618 [D] gui:718              Add view SimilarityView to GUI.
17:21:32.670 [D] gui:718              Add view WaveformView to GUI.
17:21:32.901 [D] base:337             Set state for WaveformView.
17:21:32.924 [D] gui:718              Add view CorrelogramView to GUI.
17:21:32.932 [D] base:337             Set state for CorrelogramView.
17:21:32.965 [D] gui:718              Add view ISIView to GUI.
17:21:32.977 [D] base:337             Set state for ISIView.
17:21:32.995 [D] gui:718              Add view FeatureView to GUI.
17:21:33.001 [D] base:337             Set state for FeatureView.
17:21:33.031 [D] gui:718              Add view AmplitudeView to GUI.
17:21:33.038 [D] base:337             Set state for AmplitudeView.
17:21:33.070 [D] gui:718              Add view FiringRateView to GUI.
17:21:33.074 [D] base:337             Set state for FiringRateView.
17:21:33.264 [D] gui:519              Load the geometry state.
17:21:36.810 [D] supervisor:817       Clusters selected: [1] (2)

Then the error trace is the same as before. Here is also the template.npy file (I cannot put it directly through git) : https://instituteicm-my.sharepoint.com/:u:/g/personal/anthony_pinto_icm-institute_org/EQGPeGkKugBLqWJ1RRikeWcBpt4n98QAr14rGFG_6DA7xQ?e=tsd16x

Thanks you !

Best, Anthony

[zm711]

@Antho2422

could I get template_ind.npy too? Thanks Anthony!

Zach

[Antho2422]

@zm711

There you go : https://instituteicm-my.sharepoint.com/:u:/g/personal/anthony_pinto_icm-institute_org/EZLOFQQkJ9VEszJRXv5T4BkBN6U8igNwS44AZWm-2Sk9Kg?e=hRLZaE

[zm711]

@Antho2422

So I believe the error is here. An IOError must be occurring while trying to find the templates file causes the get_templates() function to return None, rather than the templates. I'm still trying to figure out the why though. Did you use the export to phy function in SpikeInterface?

https://github.com/cortex-lab/phylib/blob/2b567cd2e2c382bd5e8926e9abd9a2f9353bfd76/phylib/io/model.py#L695-L707

[zm711]

If the file is found the debugger should print whether the templates are sparse or dense: https://github.com/cortex-lab/phylib/blob/2b567cd2e2c382bd5e8926e9abd9a2f9353bfd76/phylib/io/model.py#L720-L724

But your debug does not say this likely indicating that there was an IOError exit before hand.

[zm711]

What is the dat_path specified in your params.py. And where is the folder containing all the files (C:\users\ap\documents or something like that?). Ideally we want to see the exact io error occurring to try to figure out which function in particular is failing. Basically we have the issue narrowed down to between 2 functions either the _find_path or the _read_array.

[Antho2422]

The dat_path is the following :

dat_path = r'/network/lustre/iss02/charpier/analyses/anthony.pinto/git_lustre/priam/data/processed/golgi_v1.0/02711_2019-04-18_11-04/preprocessed/phy/1/recording.dat'

The folder containing all the files is at the same place I guess, are you talking about the phy folder ? The path is on a lustre server but I already tried it in local and I got the same issue.

[zm711]

@Antho2422

So there is an issue with np.load where if the name of a file is too long it can cause an IOError. Your file path is 152 characters which is a pretty long name (as far as Python/NumPy is concerned). Could you try to greatly shorten your file path as a test and rerun locally testing something like C:\users\name\data\recording.dat. To see if that could be causing it.

[Antho2422]

Okay I will try that asap and keep you updated thank you !

[zm711]

@Antho2422

Just wanted to see if that worked. The other option is that there are some Path issues on the SpikeInterface side. So I can propagate that up the chain if it's not a python file name length issue.

[Antho2422]

Hi @zm711,

Sorry, I was on holidays and I didn’t have time to test it yet ! I will do it asap and keep you updated of course !

Thank you !

[Antho2422]

Hi @zm711 !

Sorry for the wait ! I tried with a shorter path and it worked ! Is there any way to avoid this error even when the paths are long ?

Thank you !

[zm711]

@Antho2422

That's great!

But unfortunately no... I actually ran into this issue in a different program I was running so I opened issues myself and was told that it is just a computer+python thing. So I just empirically determined the allowable length of file and made sure not to cross that limit. (I want to say it was either 100 or 150 total characters including slashes--for my computer), but you would probably need to double check with yours and then just stay within that limit.

Happy to help!

[Antho2422]

Okay ! Thank you so much for the help !

Cheers, Anthony