Never choose a "group" value in dropdowns

CasperWA commented 2 years ago

See issues: aiidalab/aiidalab-qe#135 and aiidalab/aiidalab-qe#134.

Don't allow the application to auto-choose an index of a dropdown that corresponds to a grouping header.

csadorf commented 2 years ago

Maybe as an additional comment, the group headers are currently not visually distinguished in my environment. I'm not sure what the reason for that could be.

CasperWA commented 2 years ago

Maybe as an additional comment, the group headers are currently not visually distinguished in my environment. I'm not sure what the reason for that could be.

Could you show an example?

I'm not very diverse in my browser choices for testing, using only Chrome, so this might be the issue? Although I am using nothing else but the standard HTML tag options, I believe.

CasperWA commented 2 years ago

Can you check whether the latest version of develop works? Or should I release a new version for you to more easily test? @csadorf

csadorf commented 2 years ago

Can you check whether the latest version of develop works? Or should I release a new version for you to more easily test? @csadorf

@CasperWA Thanks for proposing a fix. I'm gonna test this, no release or prerelease needed.

csadorf commented 2 years ago

@CasperWA I've tested it and this is what it looks like on my developer machine: Screen Shot 2021-10-14 at 16 16 39

$ pip freeze | grep ipywidgets
ipywidgets==7.6.5
ipywidgets-extended==1.1.1

Selecting "General" does not crash the app anymore, but still kind of unexpected that it is even possible to select the headers. I tested it on Firefox and Chrome, same issue in both browsers.

csadorf commented 2 years ago

@CasperWA You should be able to reproduce the issue with the following steps:

In a terminal, run:

git clone https://github.com/aiidalab/aiidalab-docker-stack
cd ./aiidalab-docker-stack/
pip install -r requirements-manage.txt  # for docker-compose and click
./manage.py configure --app quantum-espresso==21.10.0b1
./manage.py --develop -v up

Open the QE app.
Select the "OPTIMADE" tab for the structure selection.
Select "Materials Cloud" as the provider.
Open the list of databases.

https://user-images.githubusercontent.com/1441208/139038845-89c68486-ae06-4ad6-bd50-cba3b022937d.mp4

unkcpz commented 2 years ago

I can reproduce this on my computer OS: Ubuntu 20.04.3 LTS Browser: Chrome and Firefox

unkcpz commented 2 years ago

Downgrade the ipywidgets-extended version to ipywidgets-extended==1.1.0 and selecting "General" DOES crash the app again.

csadorf commented 2 years ago

@unkcpz noticed that on a different deployment, the widget works as expected, however we are not able to figure out what is causing that.

Diff of pip list --format=freeze between the non-working (left) and working (right) environment:

```diff > aiida-bands-inspect==0.4.0 > aiida-bigdft==0.2.6 > aiida-codtools==2.1.0 aiida-core==1.6.4 aiida-core==1.6.4 > aiida-cp2k==1.3.0 > aiida-crystal17==0.11.0 > aiida-ddec==1.1.0 > aiida-diff==1.2.0 > aiida-fleur==1.2.1 > aiida-gaussian==1.4.1 > aiida-lammps==0.8.0 > aiida-lsmo==1.0.0 > aiida-nanotech-empa==0.4.1 > aiida-optimize==0.4.0 aiida-pseudo==0.6.3 aiida-pseudo==0.6.3 > aiida-qeq==1.0.0a1 aiida-quantumespresso==3.4.2 aiida-quantumespresso==3.4.2 > aiida-raspa==1.2.0 > aiida-sssp-workflow==0.1.0 > aiida-tbmodels==0.3.0 > aiida-tools==0.3.3 > aiida-vasp==2.1.1 > aiida-wannier90==2.0.1 > aiida-yambo==1.3.0 > aiida-zeopp==1.1.2 aiidalab==21.10.1 aiidalab==21.10.1 aiidalab-eln==0.1.1 aiidalab-eln==0.1.1 aiidalab-qe==21.10.0b1 aiidalab-qe==21.10.0b1 aiidalab-qe-workchain==1.0 aiidalab-qe-workchain==1.0 aiidalab-widgets-base==1.0.0 aiidalab-widgets-base==1.0.0 aio-pika==6.8.0 aio-pika==6.8.0 aiormq==3.3.1 aiormq==3.3.1 > alabaster==0.7.12 aldjemy==0.9.1 aldjemy==0.9.1 alembic==1.6.5 alembic==1.6.5 > aniso8601==9.0.1 ansi2html==1.6.0 ansi2html==1.6.0 appdirs==1.4.4 appdirs==1.4.4 archive-path==0.2.1 archive-path==0.2.1 argcomplete==1.12.3 argcomplete==1.12.3 argon2-cffi==21.1.0 argon2-cffi==21.1.0 arrow==1.2.0 | arrow==1.1.1 ase==3.19.3 ase==3.19.3 async-generator==1.10 async-generator==1.10 attrs==21.2.0 attrs==21.2.0 > Babel==2.9.1 backcall==0.2.0 backcall==0.2.0 backports.entry-points-selectable==1.1.0 | bands-inspect==0.3.1 bcrypt==3.2.0 bcrypt==3.2.0 binaryornot==0.4.4 binaryornot==0.4.4 black==21.9b0 < bleach==4.1.0 bleach==4.1.0 bokeh==2.4.1 | bokeh==2.4.0 bqplot==0.12.31 | bqplot==0.12.30 brotlipy==0.7.0 brotlipy==0.7.0 > bump2version==1.0.1 CacheControl==0.12.6 CacheControl==0.12.6 cachetools==4.2.4 | cached-property==1.5.2 certifi==2021.5.30 | cachetools==4.2.2 > calc-pe==1.0.1 > cclib==1.7.1 > certifi==2021.10.8 cffi==1.14.4 cffi==1.14.4 cfgv==3.3.1 | cftime==1.5.1 chardet==4.0.0 chardet==4.0.0 charset-normalizer==2.0.7 | charset-normalizer==2.0.6 cheminfopy==0.7.0 cheminfopy==0.7.0 circus==0.17.1 circus==0.17.1 click==7.1.2 click==7.1.2 click-completion==0.5.2 click-completion==0.5.2 click-config-file==0.6.0 click-config-file==0.6.0 click-spinner==0.1.10 click-spinner==0.1.10 > colorama==0.4.4 conda==4.9.2 conda==4.9.2 conda-package-handling==1.7.2 conda-package-handling==1.7.2 configobj==5.0.6 configobj==5.0.6 cookiecutter==1.7.3 cookiecutter==1.7.3 cryptography==3.3.1 cryptography==3.3.1 cycler==0.10.0 cycler==0.10.0 dacite==1.6.0 dacite==1.6.0 debugpy==1.5.0 | debugpy==1.4.3 decorator==4.4.2 decorator==4.4.2 > deepdiff==5.6.0 defusedxml==0.7.1 defusedxml==0.7.1 deprecation==2.1.0 deprecation==2.1.0 distlib==0.3.3 < Django==2.2.24 Django==2.2.24 dnspython==2.1.0 dnspython==2.1.0 > docutils==0.15.2 dulwich==0.20.25 dulwich==0.20.25 elementpath==2.3.2 elementpath==2.3.2 email-validator==1.1.3 email-validator==1.1.3 entrypoints==0.3 entrypoints==0.3 ete3==3.1.2 ete3==3.1.2 fastapi==0.65.3 fastapi==0.65.3 filelock==3.3.1 | fileupload==0.1.5 > Flask==1.1.4 > Flask-Cors==3.0.10 > Flask-RESTful==0.3.9 > fsc.export==1.2.0 > fsc.hdf5-io==1.0.3 future==0.18.2 future==0.18.2 graphviz==0.16 graphviz==0.16 identify==2.3.0 | h5py==3.5.0 > humanfriendly==10.0 idna==2.10 idna==2.10 > imagesize==1.2.0 importlib-metadata==4.6.0 importlib-metadata==4.6.0 importlib-resources==5.3.0 | importlib-resources==3.3.1 > iniconfig==1.1.1 ipydatawidgets==4.2.0 ipydatawidgets==4.2.0 ipykernel==6.4.1 ipykernel==6.4.1 ipython==7.25.0 ipython==7.25.0 ipython-genutils==0.2.0 ipython-genutils==0.2.0 ipytree==0.2.1 ipytree==0.2.1 ipywidgets==7.6.5 ipywidgets==7.6.5 ipywidgets-extended==1.1.1 | ipywidgets-extended==1.1.0 > itsdangerous==1.1.0 jedi==0.18.0 jedi==0.18.0 Jinja2==2.11.3 Jinja2==2.11.3 jinja2-time==0.2.0 jinja2-time==0.2.0 joblib==1.1.0 | joblib==1.0.1 > jsonextended==0.7.11 jsonpointer==2.1 jsonpointer==2.1 jsonref==0.2 jsonref==0.2 jsonschema==3.2.0 jsonschema==3.2.0 jupyter-client==6.1.12 jupyter-client==6.1.12 jupyter-core==4.8.1 jupyter-core==4.8.1 jupyterlab-widgets==1.0.2 jupyterlab-widgets==1.0.2 kiwipy==0.7.4 kiwipy==0.7.4 kiwisolver==1.3.1 kiwisolver==1.3.1 lark-parser==0.12.0 lark-parser==0.12.0 lockfile==0.12.2 lockfile==0.12.2 > lxml==4.6.3 Mako==1.1.4 Mako==1.1.4 Markdown==3.3.4 Markdown==3.3.4 MarkupSafe==2.0.1 MarkupSafe==2.0.1 > masci-tools==0.6.2 matplotlib==3.4.2 matplotlib==3.4.2 matplotlib-inline==0.1.2 matplotlib-inline==0.1.2 > mendeleev==0.7.0 mistune==0.8.4 mistune==0.8.4 monty==2021.6.10 monty==2021.6.10 more-itertools==8.10.0 more-itertools==8.10.0 mpmath==1.2.1 mpmath==1.2.1 msgpack==1.0.2 msgpack==1.0.2 multidict==5.1.0 multidict==5.1.0 mypy-extensions==0.4.3 < nbconvert==5.6.1 nbconvert==5.6.1 nbformat==5.1.3 nbformat==5.1.3 nest-asyncio==1.4.3 nest-asyncio==1.4.3 > netCDF4==1.5.7 networkx==2.5.1 networkx==2.5.1 nglview==3.0.3 nglview==3.0.3 nodeenv==1.6.0 < notebook==6.4.4 notebook==6.4.4 numpy==1.21.1 numpy==1.21.1 olefile==0.46 olefile==0.46 optimade==0.16.4 | optimade==0.16.6 optimade-client==2021.8.24 optimade-client==2021.8.24 > ordered-set==4.0.2 packaging==20.9 packaging==20.9 palettable==3.3.0 palettable==3.3.0 pamqp==2.3.0 pamqp==2.3.0 pandas==1.3.1 pandas==1.3.1 pandocfilters==1.5.0 pandocfilters==1.5.0 paramiko==2.7.2 paramiko==2.7.2 > parsevasp==2.0.1 parso==0.8.2 parso==0.8.2 pathspec==0.9.0 | pathlib2==2.3.6 pathtools==0.1.2 pathtools==0.1.2 pbr==5.6.0 pbr==5.6.0 > periodictable==1.6.0 pexpect==4.8.0 pexpect==4.8.0 pgsu==0.2.0 pgsu==0.2.0 pickleshare==0.7.5 pickleshare==0.7.5 Pillow==8.3.1 Pillow==8.3.1 Pint==0.16.1 Pint==0.16.1 pip==21.3 pip==21.3 platformdirs==2.4.0 < plotly==5.0.0 plotly==5.0.0 > pluggy==1.0.0 plumpy==0.19.0 plumpy==0.19.0 poyo==0.5.0 poyo==0.5.0 pre-commit==2.15.0 < prometheus-client==0.11.0 prometheus-client==0.11.0 prompt-toolkit==3.0.19 prompt-toolkit==3.0.19 psutil==5.8.0 psutil==5.8.0 psycopg2-binary==2.8.6 psycopg2-binary==2.8.6 ptyprocess==0.7.0 ptyprocess==0.7.0 > py==1.10.0 pycairo==1.20.1 pycairo==1.20.1 PyCifRW==4.4.3 PyCifRW==4.4.3 pycosat==0.6.3 pycosat==0.6.3 pycparser==2.20 pycparser==2.20 pydantic==1.8.2 pydantic==1.8.2 > pydata-sphinx-theme==0.4.3 > pyfiglet==0.8.post1 Pygments==2.9.0 Pygments==2.9.0 pymatgen==2022.0.9 | pyhull==2015.2.1 > pyiast==1.4.3 > pymatgen==2020.12.3 PyMySQL==0.9.3 PyMySQL==0.9.3 PyNaCl==1.4.0 PyNaCl==1.4.0 pyOpenSSL==20.0.1 pyOpenSSL==20.0.1 pyparsing==2.4.7 pyparsing==2.4.7 pyrsistent==0.17.3 pyrsistent==0.17.3 pysmiles==1.0.1 pysmiles==1.0.1 PySocks==1.7.1 PySocks==1.7.1 > pytest==6.2.5 python-dateutil==2.8.2 python-dateutil==2.8.2 python-editor==1.0.4 python-editor==1.0.4 > python-memcached==1.59 python-slugify==5.0.2 python-slugify==5.0.2 pythreejs==2.3.0 pythreejs==2.3.0 pytojcamp==0.1.0 pytojcamp==0.1.0 pytray==0.3.2 pytray==0.3.2 pytz==2019.3 pytz==2019.3 PyYAML==5.1.2 PyYAML==5.1.2 pyzmq==22.1.0 pyzmq==22.1.0 qe-tools==2.0.0rc2 qe-tools==2.0.0rc2 reentry==1.3.2 reentry==1.3.2 regex==2021.10.8 < requests==2.26.0 requests==2.26.0 requests-cache==0.5.2 requests-cache==0.5.2 requests-mock==1.8.0 requests-mock==1.8.0 rfc3987==1.3.8 rfc3987==1.3.8 ruamel.yaml==0.16.13 ruamel.yaml==0.16.13 ruamel.yaml.clib==0.2.2 ruamel.yaml.clib==0.2.2 ruamel-yaml-conda==0.15.80 ruamel-yaml-conda==0.15.80 scikit-learn==0.24.2 scikit-learn==0.24.2 scipy==1.7.0 scipy==1.7.0 > seaborn==0.11.2 seekpath==1.9.7 seekpath==1.9.7 Send2Trash==1.8.0 Send2Trash==1.8.0 setuptools==49.6.0.post20210108 setuptools==49.6.0.post20210108 shellingham==1.4.0 shellingham==1.4.0 shortuuid==1.0.1 shortuuid==1.0.1 simplejson==3.17.2 simplejson==3.17.2 six==1.15.0 six==1.15.0 spglib==1.16.1 | snowballstemmer==2.1.0 > spglib==1.16.2 > Sphinx==3.2.1 > sphinx-copybutton==0.3.3 > sphinx-notfound-page==0.7.1 > sphinx-panels==0.5.2 > sphinxcontrib-applehelp==1.0.2 > sphinxcontrib-details-directive==0.1.0 > sphinxcontrib-devhelp==1.0.2 > sphinxcontrib-htmlhelp==2.0.0 > sphinxcontrib-jsmath==1.0.1 > sphinxcontrib-qthelp==1.0.3 > sphinxcontrib-serializinghtml==1.1.5 > sphinxext-rediraffe==0.2.7 SQLAlchemy==1.3.24 SQLAlchemy==1.3.24 SQLAlchemy-Utils==0.36.8 SQLAlchemy-Utils==0.36.8 sqlparse==0.4.1 sqlparse==0.4.1 starlette==0.14.2 starlette==0.14.2 strict-rfc3339==0.7 strict-rfc3339==0.7 > subprocess32==3.5.4 sympy==1.8 sympy==1.8 tabulate==0.8.9 tabulate==0.8.9 tenacity==7.0.0 tenacity==7.0.0 terminado==0.12.1 terminado==0.12.1 testpath==0.5.0 testpath==0.5.0 text-unidecode==1.3 text-unidecode==1.3 threadpoolctl==3.0.0 | threadpoolctl==2.2.0 toml==0.10.2 toml==0.10.2 tomli==1.2.1 < tornado==6.1 tornado==6.1 tqdm==4.56.1 tqdm==4.56.1 traitlets==5.0.5 traitlets==5.0.5 traittypes==0.2.1 traittypes==0.2.1 typed-ast==1.4.3 | typing-extensions==3.10.0.2 typing-extensions==3.10.0.0 < tzlocal==2.1 tzlocal==2.1 uncertainties==3.1.5 uncertainties==3.1.5 upf-to-json==0.9.3 upf-to-json==0.9.3 urllib3==1.26.3 urllib3==1.26.3 Vapory==0.1.2 Vapory==0.1.2 virtualenv==20.8.1 | voluptuous==0.12.2 watchdog==0.10.4 watchdog==0.10.4 wcwidth==0.2.5 wcwidth==0.2.5 webcolors==1.11.1 webcolors==1.11.1 webencodings==0.5.1 webencodings==0.5.1 > Werkzeug==1.0.1 wheel==0.36.2 wheel==0.36.2 widget-bandsplot==0.2.5 | widget-bandsplot==0.2.4 widget-periodictable==2.1.5 widget-periodictable==2.1.5 widgetsnbextension==3.5.1 widgetsnbextension==3.5.1 wrapt==1.11.2 wrapt==1.11.2 xmlschema==1.8.1 xmlschema==1.8.1 yarl==1.6.3 yarl==1.6.3 zipp==3.4.1 zipp==3.4.1 ```

CasperWA commented 2 years ago

Hmm... this is very peculiar. So to me this might come down to two things, and both things might be wrong for where to go to try and find the issue:

It's an OS/browser support related. The idea here is that it might be caused by some incompatibilities or short-comings (uninstalled fluff plugins) for a bare-bones linux OS for any given browser.
It's an installation issue with the ipywidgets-extended package into the Jupyter environment. It's not uncommon that a widget is not properly installed as a notebook extension (nbextension). It could be that the installation protocol for the package isn't rigorous enough and doesn't find the proper place to install itself to enable it as a proper extension or that it doesn't have the rights upon installation to place itself in some critical Jupyter locations? Using virtuelenv this seems to work fine, it uses the nbconfig file under the respective virtual environment (e.g., ~/.venvs/ipywidgets/etc/jupyter/nbconfig, where ipywidgets is the name of the venv). However, it may be that this fails under other conditions?

The latter is what I'd check first, however, usually if they're not loaded properly it just doesn't show the widget at all. It simply doesn't show up. But it may still be that this is the issue and we're just seeing a different error than normal... It's all quite confusing to be honest 😅 😟

csadorf commented 2 years ago

* It's an OS/browser support related.
  The idea here is that it might be caused by some incompatibilities or short-comings (uninstalled fluff plugins) for a bare-bones linux OS for any given browser.

I highly doubt that, since we were able to reproduce the issues on Mac OS-X/Linux with both Firefox and Chrome and on top of that, it did work fine in one of the deployments using the same system. So I would say it's almost impossible to be OS or browser-related.

csadorf commented 2 years ago

@CasperWA Would you mind trying to reproduce the issue with the steps shown above (it should only take 5-10mins) and at least confirm that you observe it in your system? Maybe you can check the browser console or so and see if you recognize any of the errors or warnings?

CasperWA commented 2 years ago

I get the same thing happening when I use the approach you have mentioned above.

I tried installing the OPTIMADE Client application as well, and have been looking into the Chrome DevTools to try and see what's going on. There are several things I could point out, but I am completely unaware whether they are the culprits or not.

One is this "find-and-replace" that's missing. I don't know what it is. Find and replace widgets/code or just something else completely?

Then there's this entry, which I don't see if I simply open the "clean" Voilà edition of the client. I cannot see whether it thinks this is an error or it's just printing stuff out to the console. It might simply be the latter.

It should be noted it loads the ipywidgets_extended/extension fine. And this also makes sense, because the list of providers is the same type of widget, but it's only using the "disable options" feature.

Comparing the HTML code for the widgets in my two client editions (the one from your Docker stack and my local version) it's clear the dropdown with the databases doesn't receive any information about what to disable, the style, anything... it just seems to instantiate the widget and then... die.

I don't see other significant things in the DevTools.

One other thing I just noticed (again, might not be in any way significant) is the presence of lm-Widget in the classes of all my widgets in the local instance, while the lm-* class names are not present whatsoever in the widgets for the Docker stack edition. Now, this might simply be a Voilà thing that it adds to the classes. But it might also be a Jupyter versioning / ipywidgets versioning thing. But as you listed the pip diff above.. well... it's just weird, and maybe just a Voilà-specific thing and doesn't actually have anything to do with this issue.

My reasoning here is that I'm trying to find clues to why the information about disabling and grouping is not getting through to the dropdown widget.

csadorf commented 2 years ago

My reasoning here is that I'm trying to find clues to why the information about disabling and grouping is not getting through to the dropdown widget.

How is the information provided in the first place?

CasperWA commented 2 years ago

In any case. I unfortunately do not have more time to spend on this at this moment. If you want a (possibly) working setup, you can try to do the following:

Setup a new virtual env. using minimum Python 3.7.
Run:
```
$ pip install -U pip setuptools wheel
$ pip install optimade-client[dev]
```
Note, it might be that one needs to first run pip install jupyter. I'm unsure.
Run the server via $ optimade-client and then go to the shown URL to test it out.

CasperWA commented 2 years ago

My reasoning here is that I'm trying to find clues to why the information about disabling and grouping is not getting through to the dropdown widget.

How is the information provided in the first place?

You mean to the widget? Well it's this JS/Python-hybrid schema. I don't remember the exact timing. Whether it's on initialization of the Python object that represents the widget that the grouping is done. I don't think so though. Because the list of options needs to be updated, so this is most likely triggered when the database is chosen and can be found in the Python code. Then when/how the actual HTML is updated should be from the JS. It should be fed a variable from Python that contains a dictionary/JS object, which it then runs through to populate the dropdown options and stylize them.

csadorf commented 2 years ago

I was able to follow the code up until roughly here: https://github.com/CasperWA/voila-optimade-client/blob/a013cce2927b96d224b841a7a15004fa51159f10/optimade_client/subwidgets/provider_database.py#L336-L340 and then I get lost. Maybe you can help me understand where exactly the hand-over happens to the JS object?

CasperWA commented 2 years ago

Sure, we can have a look at some point - possibly next week.

In the mean time, it might be interesting to see whether the issue is with the extension itself or with the implementation in the client. I.e., see if you can reproduce it by just installing the ipywidgets-extended package from GitHub. Then you can test it by running the examples/examples.ipynb notebook :)

csadorf commented 2 years ago

In the mean time, it might be interesting to see whether the issue is with the extension itself or with the implementation in the client. I.e., see if you can reproduce it by just installing the ipywidgets-extended package from GitHub. Then you can test it by running the examples/examples.ipynb notebook :)

I just ran the example and it seems to fail: Screen Shot 2021-10-27 at 16 40 39

csadorf commented 2 years ago

@CasperWA Based on my last experiment it seems that the issue is with the extension, would you agree?

CasperWA commented 2 years ago

@CasperWA Based on my last experiment it seems that the issue is with the extension, would you agree?

Yes indeed. I thought I commented on this, but I must have missed pushing the button. So we should attack this issue on the side of the extension instead: CasperWA/ipywidgets-extended.

csadorf commented 2 years ago

@CasperWA Based on my last experiment it seems that the issue is with the extension, would you agree?

Yes indeed. I thought I commented on this, but I must have missed pushing the button. So we should attack this issue on the side of the extension instead: CasperWA/ipywidgets-extended.

@CasperWA Would you mind transferring https://github.com/CasperWA/voila-optimade-client/issues/369 to https://github.com/CasperWA/ipywidgets-extended? I lack permissions to do so.

csadorf commented 2 years ago

Auxilliary issue now tracked in #369 .

CasperWA / voila-optimade-client

Never choose a "group" value in dropdowns #357