search

cortex-lab / phy

phy: interactive visualization and manual spike sorting of large-scale ephys data
BSD 3-Clause "New" or "Revised" License
323 stars 158 forks source link

ValueError: attempt to get argmax of an empty sequence #1233

Closed litzj01 closed 5 months ago

litzj01 commented 1 year ago

I have tried so many installation methods and cannot get Phy to work whatsoever. Any help would be appreciated.

This is what I did for my most recent installation attempt:

  1. Downloaded the environment.yml file
  2. Edited the environment file with instructions from #1228
  3. Created the environment and followed the install instructions

conda env create -f environment.yml conda activate phy2 pip install git+https://github.com/cortex-lab/phy.git

  1. Tried to run the gui in debug mode

cd c:\Users\litzj\092223_Raw\preprocessed\82_wt_100Hz_filter\phy_folder_for_hdspikes phy template-gui params.py --debug

Then I get the following error:

16:40:13.138 [D] init:68 Start capturing exceptions. 16:40:13.155 [D] model:619 Loading spike clusters. 16:40:13.230 [D] model:569 No channel shank file found. 16:40:13.230 [D] model:692 Loading templates. 16:40:13.239 [D] model:720 Templates are sparse. 16:40:13.240 [W] model:667 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed. 16:40:13.240 [D] model:730 Loading the whitening matrix. 16:40:13.241 [D] model:434 Whitening matrix file not found. 16:40:13.246 [D] model:737 Loading the inverse of the whitening matrix. 16:40:13.331 [D] model:766 Loading features. 16:40:13.339 [D] model:781 Features are sparse. 16:40:13.340 [D] model:803 Loading template features. 16:40:13.342 [D] model:504 Load cluster_channel_group.tsv. 16:40:13.345 [D] model:504 Load cluster_group.tsv. 16:40:13.347 [D] model:504 Load cluster_si_unit_ids.tsv. 16:40:13.866 [D] context:100 Initialize joblib cache dir at C:\Users\litzj\092223_Raw\preprocessed\82_wt_100Hz_filter\phy_folder_for_hdspikes\.phy. 16:40:13.867 [D] context:101 Reducing the size of the cache if needed. 16:40:13.874 [D] base:102 Add filter high_pass. 16:40:13.875 [D] config:31 Load config file C:\Users\litzj\.phy\phy_config.py. 16:40:13.876 [D] plugin:145 Loading 0 plugins. 16:40:13.879 [D] context:209 The file C:\Users\litzj\092223_Raw\preprocessed\82_wt_100Hz_filter\phy_folder_for_hdspikes\.phy\new_cluster_id.pkl doesn't exist. 16:40:13.891 [D] context:185 Save data to C:\Users\litzj\092223_Raw\preprocessed\82_wt_100Hz_filter\phy_folder_for_hdspikes\.phy\spikes_per_cluster.pkl. 16:40:13.915 [D] gui:463 Creating GUI. 16:40:13.942 [D] state:46 Load C:\Users\litzj.phy\TemplateGUI\state.json for GUIState. 16:40:13.988 [E] init:62 An error has occurred (ValueError): attempt to get argmax of an empty sequence Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C:\Users\litzj\miniconda\envs\phy2\Scripts\phy.exe__main.py", line 7, in sys.exit(phycli()) ^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\click\core.py", line 1157, in call return self.main(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\click\core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\click\core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\click\core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\click\core.py", line 783, in invoke return callback(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\click\decorators.py", line 33, in new_func return f(get_current_context(), args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\apps__init__.py", line 159, in cli_template_gui template_gui(params_path, kwargs) File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\apps\template\gui.py", line 218, in template_gui gui = controller.create_gui() ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\apps\base.py", line 1700, in create_gui self.supervisor.attach(gui) File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\cluster\supervisor.py", line 950, in attach self._create_views( File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\cluster\supervisor.py", line 765, in _create_views gui, data=self.cluster_info, columns=self.columns, sort=sort) ^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\cluster\supervisor.py", line 925, in cluster_info return [self.get_cluster_info(cluster_id) for cluster_id in self.clustering.cluster_ids] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\cluster\supervisor.py", line 925, in return [self.get_cluster_info(cluster_id) for cluster_id in self.clustering.cluster_ids] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\cluster\supervisor.py", line 750, in get_cluster_info out[key] = func(cluster_id) ^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\apps\base.py", line 1194, in get_best_channel_label return self._get_channel_labels([self.get_best_channel(cluster_id)])[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\utils\context.py", line 154, in memcached out = f(args, kwargs) ^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\apps\base.py", line 1188, in get_best_channel channel_ids = self.get_best_channels(cluster_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\utils\context.py", line 154, in memcached out = f(args, kwargs) ^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phy\apps\template\gui.py", line 151, in get_best_channels template = self.model.get_template(template_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phylib\io\model.py", line 957, in get_template return self._get_template_sparse(template_id, unwhiten=unwhiten) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\phylib\io\model.py", line 924, in _get_template_sparse best_channel = channel_ids[np.argmax(amplitude)] ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\numpy\core\fromnumeric.py", line 1229, in argmax return _wrapfunc(a, 'argmax', axis=axis, out=out, kwds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\litzj\miniconda\envs\phy2\Lib\site-packages\numpy\core\fromnumeric.py", line 59, in _wrapfunc return bound(args, **kwds) ^^^^^^^^^^^^^^^^^^^^ ValueError: attempt to get argmax of an empty sequence

zm711 commented 1 year ago

@litzj01,

Are you using SpikeInterface? Maybe this is the problem: sparse-issue, This can come up if you have mono channel templates. If using SI, you may need to change your sparsity (note SI changed the sparsity defaults recently during waveform extraction so you would need to change at that point). Otherwise, if not using SI, which sorter are you using?

litzj01 commented 1 year ago

Thanks for your reply! I am using the Herding Spikes sorter in SpikeInterface and have 4096 channels from a biocam MEA. This is my output for running the code to check the template shapes. Does this look correct? If I do need to change the sparsity during waveform extraction, do you have any guidelines on how to do that (I'm pretty new to SI)?

sparsity_templates

zm711 commented 1 year ago

Yes that would be the correct thing to check, But it is not displaying all of your templates for me to see. If any of your templates say (58,1) that would the problem. 

for unit_id in we.unit_ids:
    template = we.get_template(unit_id)
    assert template.shape[1]!=1, "mono-channel template"

The docs for selecting how to set your sparsity are given here: sparsity docs

Basically when you run extract_waveforms it can handle any sparsity kwargs that you give it so the best explanation is here in the docstring. of the extract_waveform function. The type of sparsity settings you use is likely dependent on the type of probe you are using, so you would need to know the spacing of your channels on the biocam system.

I'm not super familiar with biocam, so it might be best to open this issue on the SpikeInterface tracker since this isn't a Phy problem, but a sparsity issue in SpikeInteface but could do something like plot_probe and show me what your probe geometry looks like.

The other option is you could extract_waveforms(...., spare=False) and start with dense waveforms to be exported into Phy and see how it looks with dense waveforms. (it is not ideal and a bit slow, but would be proof of principle that this can work).

litzj01 commented 1 year ago

Sorry, I forgot to display to full output. The lowest template is (58,8), so none of the units have a template of (58,1). The probe is 2.67 mm x 2.67 mm and each electrode is 21 µm x 21 µm with a pitch of 42 µm. I can try to open the issue with SpikeInterface as well as it seems others have had problems with this.

Probe_map

zm711 commented 1 year ago

@litzj01, cool it is definitely worth a try there. You can tag me over there too. I would supply the full error trace and the template info that your smallest template is (58,8). The plot_probe is also useful to see. Also worth linking the issue I linked above as well.

The other thing you could try first is doing with dense as I mentioned above. So just re-extract the waveforms with sparse=False and then that will let us know for sure that the sparsity is the issue or if there is another problem.

litzj01 commented 1 year ago

Thanks! I'll try extracting the dense waveforms now and let you know if it works!

litzj01 commented 1 year ago

How long would you expect the export_to_phy to take? I was able to extract the waveforms with sparse = False, and the folder ended up being 48GB which seems quite large. However, when I ran export_to_phy, it timed out after 12 hours. If you think it is still worth it, can try again with a longer timed session.

zm711 commented 1 year ago

If your dense file data is 48 gb, it might take too long to export. I would probably just raise the issue over on spikeinterface.

litzj01 commented 1 year ago

I am following up on the issues I am still having with Phy. I am working on extracting the waveforms with a different defined sparsity, but am now experiencing a new error (mentioned in https://github.com/SpikeInterface/spikeinterface/issues/2200).

I run the following line in spikeinterface to export my data to phy: sexp.export_to_phy(we_sparse, "/home/exacloud/gscratch/Sivyerlab/092123_Primate/sorted/no_filter/" + 'phy_folder_for_hdspikes', verbose=True)

Then I switch to my phy environment and run: cd /home/exacloud/gscratch/Sivyerlab/092123_Primate/sorted/no_filter/phy_folder_for_hdspikes phy template-gui params.py --debug

At some point a new window opens for a couple seconds, but it is full of static. Here is my new error:

13:48:59.771 [D] __init__:68          Start capturing exceptions.
13:49:00.002 [D] model:619            Loading spike clusters.
13:49:00.229 [D] model:569            No channel shank file found.
13:49:00.230 [D] model:692            Loading templates.
13:49:00.262 [D] model:720            Templates are sparse.
13:49:00.265 [W] model:667            Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed.
13:49:00.265 [D] model:730            Loading the whitening matrix.
13:49:00.265 [D] model:434            Whitening matrix file not found.
13:49:00.282 [D] model:737            Loading the inverse of the whitening matrix.
13:49:00.861 [D] model:766            Loading features.
13:49:00.917 [D] model:781            Features are sparse.
13:49:00.918 [D] model:803            Loading template features.
13:49:00.921 [D] model:504            Load `cluster_group.tsv`.
13:49:00.931 [D] model:504            Load `cluster_channel_group.tsv`.
13:49:00.934 [D] model:504            Load `cluster_si_unit_ids.tsv`.
qt.qpa.xcb: X server does not support XInput 2
failed to get the current screen resources
qt.qpa.gl: QXcbConnection: Failed to initialize GLX
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled
13:49:15.649 [D] context:100          Initialize joblib cache dir at `/home/exacloud/gscratch/Sivyerlab/092123_Primate/sorted/no_filter/phy_folder_for_hdspikes/.phy`.
13:49:15.698 [D] context:101          Reducing the size of the cache if needed.
13:49:15.748 [D] base:102             Add filter `high_pass`.
13:49:15.749 [D] config:31            Load config file `/home/users/litzj/.phy/phy_config.py`.
13:49:15.751 [D] plugin:145           Loading 0 plugins.
13:49:15.757 [D] context:209          The file `/home/exacloud/gscratch/Sivyerlab/092123_Primate/sorted/no_filter/phy_folder_for_hdspikes/.phy/new_cluster_id.pkl` doesn't exist.
13:49:15.889 [D] context:185          Save data to `/home/exacloud/gscratch/Sivyerlab/092123_Primate/sorted/no_filter/phy_folder_for_hdspikes/.phy/spikes_per_cluster.pkl`.
13:49:16.086 [D] gui:463              Creating GUI.
13:49:16.908 [D] state:46             Load /home/users/litzj/.phy/TemplateGUI/state.json for GUIState.
13:49:22.452 [D] gui:718              Add view ClusterView to GUI.
13:49:22.554 [D] gui:718              Add view SimilarityView to GUI.
13:49:22.713 [D] gui:718              Add view WaveformView to GUI.
13:49:22.795 [D] base:337             Set state for WaveformView.
13:49:22.826 [D] gui:718              Add view CorrelogramView to GUI.
13:49:22.829 [D] base:337             Set state for CorrelogramView.
13:49:22.883 [D] gui:718              Add view ISIView to GUI.
13:49:22.887 [D] base:337             Set state for ISIView.
13:49:22.922 [D] gui:718              Add view FeatureView to GUI.
13:49:22.925 [D] base:337             Set state for FeatureView.
13:49:22.972 [D] gui:718              Add view AmplitudeView to GUI.
13:49:22.976 [D] base:337             Set state for AmplitudeView.
13:49:23.030 [D] gui:718              Add view FiringRateView to GUI.
13:49:23.034 [D] base:337             Set state for FiringRateView.
13:49:44.803 [D] gui:718              Add view TraceView to GUI.
13:49:44.931 [D] base:337             Set state for TraceView.
13:49:44.931 [D] base:494             Set color scheme to depth.
13:49:55.464 [D] base:1626            Adding default color schemes to TraceView.
13:50:09.233 [D] gui:718              Add view ProbeView to GUI.
13:50:09.366 [D] base:337             Set state for ProbeView.
13:50:09.378 [W] gui:671              Could not create view TemplateFeatureView.
13:50:09.430 [D] gui:718              Add view ClusterScatterView to GUI.
13:50:09.433 [D] base:337             Set state for ClusterScatterView.
13:50:09.433 [D] base:494             Set color scheme to firing_rate.
13:50:09.436 [D] base:1626            Adding default color schemes to ClusterScatterView.
13:50:09.836 [D] gui:519              Load the geometry state.
qt.qpa.xcb: QXcbConnection: XCB error: 1 (BadRequest), sequence: 165, resource id: 90, major code: 130 (Unknown), minor code: 47
qt.qpa.xcb: QXcbConnection: XCB error: 170 (Unknown), sequence: 180, resource id: 90, major code: 146 (Unknown), minor code: 20
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled

QOpenGLWindow::beginPaint: Failed to create context
QOpenGLWindow::beginPaint: Failed to make context current
QOpenGLFunctions created with non-current context
Segmentation fault (core dumped)
zm711 commented 1 year ago

@litzj01,

Sorry for making you bounce around, but I just want to keep the errors in their own separate streams.

It looks like you're on linux. Is that true? Which distro?

Could you also give me a list of the packages you have in the phy with a conda list inside of the activate phy environment. Based on your first comment you should be in a conda env and I just want to see which python, numpy etc you have.

I'm a little worried about this:

X server does not support XInput 2
litzj01 commented 1 year ago

I'm using a remote ssh server via PuTty (CentOS Linux Version 7; Linux Kernel 3.10.0-1160.66.1.el7.x86_64). I also have X11 forwarding enabled through Xming.

I'll attach the packages as a txt file Phy_packages_exacloud.txt

zm711 commented 1 year ago

The packages look okay, but I think that the forwarding through X11 with Xming is the problem. 

X server does not support XInput 2

leads to these errors:

qt.qpa.gl: QXcbConnection: Failed to initialize GLX
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib64/dri)
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled

I think what would be best is if you could try to install phy locally on a computer and just see if you can open the data on your computer. Since you're using putty I assume you have a windows computer. Phy installs fine on Windows as well (just not great on macs). so if you could test one dataset locally that would let us know if this is due to your forwarding.

litzj01 commented 1 year ago

I downloaded the Phy export from PuTty on my local computer and tried opening it in Phy and it worked!

I believe my initial issue with the ValueError: attempt to get argmax of an empty sequence was related to my sparsity setting which was fixed by adjusting the radius during waveform extraction.

My current issue withX server does not support XInput 2 is likely related to Xming and my ssh server. I am working with the support team on that to see if they are able to help fix the issue (unless you have any suggestions). I'll post an update on the solution if it ends up being relevant.

zm711 commented 1 year ago

@litzj01,

That's great to hear. So SpikeInterface is working and Phy is working!

As far as the Xming and ssh, I do not have additional advice. Spikeinterface does have an interface with sortingview which you might want to check out. It creates a Phy like interface that can be opened in the browser (a little less feature rich than Phy right now --no split, no instant PC views), but you could analyze your data on the server, export to sortingview, copy the browser link and sort on your laptop (it is super light), export a json of your sorting back to the server and then do the curation on the server.

Or you do Phy locally, edit all the files and then when done you send them back to the server to do your post analysis? But I'm just brainstorming now.