Unable to open Phy due to QT error

[jonahpearl]

Hi there -- I'm encountering an unusual QT error when trying to open phy. AFAIK, nothing changed between me having phy working and this error, but this is a shared computer so there may be deep-rooted issues in the conda install.

The command I'm running, in the phy2 conda environment, is: phy template-gui D:\...\path\to\params.py.

The exact error message is: [16340:16344:0621/164647.773:ERROR:extension_system_qt.cpp(121)] Failed to parse extension manifest.

I enabled QT debugging so I could get a verbose output, it doesn't seem very helpful to me (error message is on the last line, everything before seems ok): phy debug qt log.txt

I then tried the following: -- run on a params file that worked in the past (only a few weeks ago!!) --> same error -- uninstall the env and re-install from the command given in the README --> same error -- uninstall the env and re-install from the yaml file --> same error. -- pip install PyQtWebEngine --> same error -- pip uninstall pyqt5 pyqt5-tools; pip install pyqt5 pyqt5-tools pyqt5.sip --> different error, QWidget: Must construct a QApplication before a QWidget. Log still doesn't seem that informative: phy debug qt log after qt5sip.txt

Again, this is on a shared Windows 10 machine, so it's entirely possible there is a weird path issue going on that I can't see. Environment info: phy2_env_list.txt

I will probably start un-installing / re-installing anaconda next week unless someone has a good idea to debug this.

Thanks!

[zm711]

@jonahpearl,

One quick easy check would be are you sure you aren't accidentally escaping. In the log you have \v in your file name which could introduce a tab. I haven't seen how well Phy sanitizes Windows paths. A quick would be to cd into the folder instead.

cd/d D:\...\path\to\phy
phy template-gui params.py

Not sure which terminal you're using so I gave an example for the cmd prompt/anaconda prompt.

I have never seen that error. Could you post the phy debug information.

so

phy template-gui params.py --debug

[jonahpearl]

Gave it a shot -- no luck. Here is the phy debug from that try, rather than the QT logs. I'll do a re-install so I can repro the first error and see what the phy debug looks like. Edit: it looks the same but with the initial error on the final line. phy debug qt log after qt5sip in dir.txt

[zm711]

Could you just reinstall the environment from the yaml file, but limit the numpy to < 2.0. So use this as the yaml:

name: phy2
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - pip
  - git
  - numpy<2.0
  - matplotlib
  - scipy
  - h5py
  - pyqt
  - pyopengl
  - pyqtwebengine
  - pytest
  - qtconsole
  - requests
  - responses
  - traitlets
  - dask
  - cython
  - pillow
  - scikit-learn
  - joblib
  - pip:
    - git+https://github.com/cortex-lab/phy.git

Phy is in maintenance mode rather than dev mode, so I haven't had time to try to patch things for Numpy 2.0 yet. With the QT issue I'm worried the install got nuked. So uninstall and reinstall with the above yaml info.

[jonahpearl]

Thanks Zach. No luck with numpy 1.26.4 py311h0b4df5a_0 conda-forge.

[zm711]

2 questions:

Same QT error?

What is the input? Is this native KS? export_to_phy? (and which KS if it from KS).

[jonahpearl]

Same QT error?

Yeah

What is the input? Is this native KS? export_to_phy? (and which KS if it from KS).

This is from spikeinterface's export_to_phy, yes. I haven't had any issues before with this output, aside from much more minor phy issues, like some malformed PC components for certain units.

I used the above yaml file to install phy on a different Windows computer in the lab, and it was able to open some of my data no problem, same as I was able to on this troublesome computer some weeks ago. I verified that the conda list outputs match exactly between the two computers, and that I was using the exact same set of data, but nonetheless the issue persists. I'm going to call it a day and try again on Monday. Thanks!

[zm711]

Could you try to recreate the environment with #1289? I want to see if we limit numpy at install and in the requirements.txt if that helps. Feel free to edit the env name in the yaml if you want to keep this current env to try to test some more. Based on your conda list check this seems like less of a problem, but I am seeing some cases of packages getting out of sync as people pip install -U or update some packages without the others being updated.

[jbwallace123]

Following the issue since we are having the same issue. I have tried: installing with the yaml and fixing the numpy version -- same issue, installing/uninstalling PyQT libraries as @jonahpearl has tried -- same error. Building environment from scratch multiple times with developer instructions -- same error.

We are using a windows 11 machine and spike interface for sorting via KS2.5

[zm711]

Could you post your

phy templateg-gui params.py --debug

Based on some troubleshooting in #1287, I'm starting to wonder if we need to fix up the export_to_phy function in spikeinterface. If I can see your actual error it might help point me in the right direction.

[jbwallace123]

Unfortunately, it's not too informative:

(phy2) PS D:\Shijia\ecephys_NPfile_to_analyze\20240625_M212_4A25_g0\kilosort2.5_output\sorter_output> phy template-gui params.py --debug
12:55:41.257 [D] __init__:68          Start capturing exceptions.
12:55:41.441 [W] model:603            Unreferenced clusters found in templates (generally not a problem)
12:55:41.572 [D] model:619            Loading spike clusters.
12:55:41.706 [W] model:625            Unreferenced clusters found in spike_clusters (generally not a problem)
12:55:41.836 [D] model:569            No channel shank file found.
12:55:41.836 [D] model:692            Loading templates.
12:55:41.860 [D] model:724            Templates are dense.
12:55:41.868 [W] model:667            Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed.
12:55:41.868 [D] model:730            Loading the whitening matrix.
12:55:41.868 [D] model:737            Loading the inverse of the whitening matrix.
12:55:41.876 [D] model:766            Loading features.
12:55:41.876 [D] model:781            Features are sparse.
12:55:41.876 [D] model:803            Loading template features.
12:55:41.876 [D] model:813            Template features are sparse.
12:55:41.876 [D] model:504            Load `cluster_Amplitude.tsv`.
12:55:41.885 [D] model:504            Load `cluster_ContamPct.tsv`.
12:55:41.889 [D] model:504            Load `cluster_group.tsv`.
12:55:41.889 [D] model:504            Load `cluster_KSLabel.tsv`.
12:55:42.138 [D] context:100          Initialize joblib cache dir at `D:\Shijia\ecephys_NPfile_to_analyze\20240625_M212_4A25_g0\kilosort2.5_output\sorter_output\.phy`.
12:55:42.138 [D] context:101          Reducing the size of the cache if needed.
12:55:42.144 [D] base:102             Add filter `high_pass`.
12:55:42.144 [D] config:31            Load config file `C:\Users\janet\.phy\phy_config.py`.
12:55:42.144 [D] plugin:146           Loading 0 plugins.
12:55:42.144 [D] context:209          The file `D:\Shijia\ecephys_NPfile_to_analyze\20240625_M212_4A25_g0\kilosort2.5_output\sorter_output\.phy\new_cluster_id.pkl` doesn't exist.
12:55:42.214 [D] context:185          Save data to `D:\Shijia\ecephys_NPfile_to_analyze\20240625_M212_4A25_g0\kilosort2.5_output\sorter_output\.phy\spikes_per_cluster.pkl`.
12:55:42.259 [D] gui:463              Creating GUI.
12:55:42.270 [D] state:46             Load C:\Users\janet\.phy\TemplateGUI\state.json for GUIState.
QWidget: Must construct a QApplication before a QWidget

[jonahpearl]

Gave it a shot -- no luck. Here is the phy debug from that try, rather than the QT logs. I'll do a re-install so I can repro the first error and see what the phy debug looks like. Edit: it looks the same but with the initial error on the final line. phy debug qt log after qt5sip in dir.txt

Zm, this is my output from that command ^^. Looks the same as JBW's, crash right after the line

17:21:37.812 [D] state:46             Load D:\Jonah\phy_pcs\.phy\state.json for GUIState.

Interesting that you guys are having this exact problem too...bit of a coincidence that our labs are right next door...? Have you tried this on any other Windows computers in your lab? Because I was able to install + run phy on a rig computer (in GB) with no issues at all.

[zm711]

I'm wondering if one of the qt packages updated to deal with the numpy change and now I have to figure out how to downgrade it to make this work. That's my guess. Either 1) spikeinterface change needs to be made (I can't imagine why though. We haven't changed that export function so it should be fine) or 2) the packages updated and I need to pin the environment.yml and the requirements.txt files to protect people.

Thanks both for bearing with me. I'm in the middle of some wet lab experiments so I haven't been able to put my full effort to this yet :(

[jbwallace123]

@jonahpearl It works on the analysis machine right next to the error-throwing one... no clue as to what is going on! The working one is Windows 10 and the non-working one is Windows 11. I think the frustrating part is it was working on the Windows 11 machine and just started to throw errors this week.

Thanks, @zm711. We appreciate your efforts!

[jonahpearl]

Zm I know what you mean (seems like it must be a package / version issue), but in both of these cases, we had a computer + env that was working perfectly fine, until it wasn't, seemingly without any kind of env update. I installed phy on a different computer with the same OS and had conda list's that matched exactly (as in, I put them into VS Code and ran a diff). The only difference I can think of betw the two computers is 1) they're on different networks, 2) the one that worked probably isn't running Windows updates, 3) they might be using different env solvers for conda (mamba, on the broken one, vs the old default on the one that's working). I'm gonna try rolling back some Windows updates.

FWIW, all of the discussion above from me was installing from scratch -- I hadn't changed numpy in the reqs file, I'll try that but I'd be surprised if that fixed it, given that I ended up with numpy<2.0 anyways.

[zm711]

Great point at @jonahpearl. I had forgotten you mentioned that. Do you have some sort of Windows defender? Maybe qt functionality is being blocked on certain computers (and maybe it got turned on in an update for Windows 11? @jbwallace123 ?).

Someone else had a gui -json issue and installing with the conda yaml worked for them (if we limit numpy < 2.0). I have a PR #1289 that you could try to use? If you want the conda resolver to see if that fixes it.

[jonahpearl]

Brief update before i go do something else: I followed the dev install instructions exactly (cloned the repos, installed reqs and reqs-dev, then did pip install -e . --upgrade from phylib) and set numpy to <2.0, and now it works fine. The list of envs is quite different, and also everything is from pypi rather than conda forge. No clue what the reason for the difference is, but JBW, I know you said you tried following the developer instructions, but maybe try again with numpy < 2.0?

[zm711]

Yeah part of the problem is when you mix conda and pip they don't actually communicate so you can easily break an env. So people really need to do either just conda install/conda create OR pip install (even if inside a conda env) and not mix a bunch of conda create then pip install then pip uninstall etc....

[jbwallace123]

Hi @jonahpearl and @zm711, thank you both for the input!

This is what I did that seems to fix it:

• Ran windows updates and checked the firewall
• Removed my old environment and followed the developer instructions. But first, I edited the requirements to numpy<2.0.
• Created a new phy2 env with python==3.11
• The GUI now opens!
• Editing my previous comment, the data was on the wrong drive (HDD rather than SSD)! All good now. Thank you both!

[zm711]

Perfect, so this is telling me that the environment.yaml needs to be tweaked because conda is installing something that pip is then messing with breaking all the code. I'll try to make sure the requirements.txt and the yaml are as synced as can be. (It could be that pypi has the package before conda-forge or something like that).

This is good to know and gives me a better starting point!

[jonahpearl]

Closing the loop on the windows update bit...I reinstalled the update and had no issues. So hopefully that wasn't it.

[zm711]

Thanks Jonah. I tried one tweak on the conda yaml. there was a missing limit on pyopengl. If either of you has bandwidth to quickly test that for me and just see if the conda install works now that would be awesome. It would just be download the yaml change the name to like phy2_test and see if it works. You would have to grab the yaml from my PR #1289. (numpy limit already added).

also sometimes computers just need a shutdown to clear stuff out so maybe that was all the computer needed (+ the numpy limit :) )

[jonahpearl]

I tried that -- no luck, same error. Here is the conda list output from the phy2_test one that just failed, and the pip list and 'conda list` outputs from the phy-dev one that works.

phy-dev_piplist.txt phy2_test_list.txt phy-dev_condalist.txt

[zm711]

Thanks @jonahpearl when I get some time I'll have to do some more careful troubleshooting to see which package is getting messed up. I'll probably parse your txt files to see if I can find anything.