search

Degiacomi-Lab / molearn_notebook

Jupyter notebooks demonstrating molearn usage
GNU General Public License v3.0
3 stars 0 forks source link

[JOSS] Some problems with Notebooks #3

Closed RMeli closed 1 year ago

RMeli commented 1 year ago

2 is fixed, but unfortunately I'm still experiencing some issues with notebooks.

Issues

minimal_example

The GUI is sometimes unresponsive and sometimes I can't click on the points in latent space to create the path. Is this a known issue or it is just down to my setup somehow?

Notes:

molearn_GUI

Failure In [6]

Fails at In [6]

data_train, data_valid = data.split(manual_seed=25)
MA.set_dataset("training", data_train)
MA.set_dataset("validation", data_test)

with the following error:

NameError                                 Traceback (most recent call last)
Cell In[6], line 3
      1 data_train, data_valid = data.split(manual_seed=25)
      2 MA.set_dataset("training", data_train)
----> 3 MA.set_dataset("validation", data_test)

NameError: name 'data_test' is not defined

Changing data_test into data_valid solves the issue and allows to continue.

Failure In [14]

landscape_target_rmsd, xaxis, yaxis = MA.scan_error_from_target(MA.test_set[0])

with the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[14], line 1
----> 1 landscape_target_rmsd, xaxis, yaxis = MA.scan_error_from_target(MA.test_set[0])

AttributeError: 'MolearnAnalysis' object has no attribute 'test_set'

molearn_analysis

The kernel crashes for me at In [5]:

The kernel appears to have died. It will restart automatically.

Suggestion

This is certainly not required for JOSS, but if you want to make sure that the notebooks run one possible solution would be to automatically extract the code from the notebook with nbconvert and run it in CI (using GitHub actions). This would help to automatically spot issues (such as #2), especially if the API of molearn changes in the future.

https://github.com/openjournals/joss-reviews/issues/5523

RMeli commented 1 year ago

molearn_GUI

Playing around a bit more with the GUI I encountered the following:

I'm starting to suspect a problem with my own installation, but I can't quite pinpoint it yet.

RMeli commented 1 year ago

I'm starting to suspect a problem with my own installation, but I can't quite pinpoint it yet.

Looks like I should have read the FAQ more carefully.... =) I'll try to pin ipywidgets=7.7.1, since I'm using 8.0.6.

degiacom commented 1 year ago

Thank you for noting these bugs! I have fixed the two issues in molearn_GUI, the second one (in [14]) also required a small bugfix to molearn itself. About the issues with the GUI, as you noticed, everything should work correctly with ipywidgets=7.7.1.

RMeli commented 1 year ago

I installed the latest version of molearn (see https://github.com/Degiacomi-Lab/molearn/issues/6#issuecomment-1626012882) and the progress bars are a truly fantastic QOL improvement, thanks for adding them! The molearn_GUI notebook is running great now.

RMeli commented 1 year ago

Unfortunately molearn_analysis still crashes for me. I looked into it and it crashes when using nglview. Removing w = nv.show_mdanalysis(universe_training) and friends allows to continue.

I had some random failures during the encoding In[12], but I think it's because the notebook is pushing the limit of what can be achieved with 64GB of RAM. I see peaks of 90%/95% memory usage, so the crash might be spurious (due to other things running).

Then, it crashed again at In[17], where memory usage was 100%:

Screenshot from 2023-07-07 22-40-03

Out of curiosity, what are the specs of the system where you are usually running the notebooks? Is it a workstation or an HPC cluster?

Unfortunately for the GUI, reverting to ipywidgets=7.7.1 doesn't solve the issue and I don't know why it worked the first time around and now it doesn't. @degiacom Would you mind sharing the result of conda list --explicit so that I can build the same conda environment as you?

degiacom commented 1 year ago

About the random crashes at In[12], something odd is definitely going on, I am successfully running the notebook with the data provided in the repo on my laptop with 8 GB RAM. One thing I would like to verify, is whether this crash still takes place if you reduce the number of parallel processes running, i.e., setting MA.processes = 2 in In[7].

I am investigating the problem with nglview. I am currently running on a new laptop with a fresh environment, and I can confirm that (while calculations above run as intended), the 3D protein visualisation and traces on the latent space are not visible. This is the environment, could you verify if this at least solves the memory problem?

https://repo.anaconda.com/pkgs/main/win-64/blas-1.0-mkl.conda
https://repo.anaconda.com/pkgs/main/win-64/ca-certificates-2023.05.30-haa95532_0.conda
https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/font-ttf-inconsolata-3.000-h77eed37_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/font-ttf-source-code-pro-2.038-h77eed37_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/font-ttf-ubuntu-0.83-hab24e00_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/intel-openmp-2023.1.0-h57928b3_46319.conda
https://conda.anaconda.org/conda-forge/win-64/libexpat-2.5.0-h63175ca_1.conda
https://conda.anaconda.org/conda-forge/win-64/msys2-conda-epoch-20160418-1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pandoc-3.1.3-h57928b3_0.conda
https://conda.anaconda.org/conda-forge/noarch/pybind11-abi-4-hd8ed1ab_3.tar.bz2
https://conda.anaconda.org/pytorch/noarch/pytorch-mutex-1.0-cpu.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/tzdata-2023c-h04d1e81_0.conda
https://conda.anaconda.org/conda-forge/win-64/ucrt-10.0.22621.0-h57928b3_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/winpty-0.4.3-4.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/expat-2.5.0-h63175ca_1.conda
https://conda.anaconda.org/conda-forge/noarch/fonts-conda-forge-1-0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/m2w64-gmp-6.1.0-2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/m2w64-libwinpthread-git-5.0.0.4634.697f757-2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/vc14_runtime-14.36.32532-hfdfe4a8_17.conda
https://conda.anaconda.org/conda-forge/noarch/fonts-conda-ecosystem-1-0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/m2w64-gcc-libs-core-5.3.0-7.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/vs2015_runtime-14.36.32532-h05e6639_17.conda
https://conda.anaconda.org/conda-forge/win-64/m2w64-gcc-libgfortran-5.3.0-6.tar.bz2
https://repo.anaconda.com/pkgs/main/win-64/vc-14.2-h21ff451_1.conda
https://conda.anaconda.org/conda-forge/win-64/bzip2-1.0.8-h8ffe710_4.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/cudatoolkit-11.8.0-h09e9e62_11.conda
https://conda.anaconda.org/conda-forge/win-64/fmt-9.1.0-h181d51b_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/icu-72.1-h63175ca_0.conda
https://conda.anaconda.org/conda-forge/win-64/khronos-opencl-icd-loader-2023.04.17-h64bf75a_0.conda
https://conda.anaconda.org/conda-forge/win-64/lerc-4.0.0-h63175ca_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libaec-1.0.6-h63175ca_1.conda
https://conda.anaconda.org/conda-forge/win-64/libbrotlicommon-1.0.9-hcfcfb64_9.conda
https://conda.anaconda.org/conda-forge/win-64/libdeflate-1.18-hcfcfb64_0.conda
https://conda.anaconda.org/conda-forge/win-64/libffi-3.4.2-h8ffe710_5.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libiconv-1.17-h8ffe710_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libjpeg-turbo-2.1.5.1-hcfcfb64_0.conda
https://conda.anaconda.org/conda-forge/win-64/libogg-1.3.4-h8ffe710_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libsodium-1.0.18-h8d14728_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libsqlite-3.42.0-hcfcfb64_0.conda
https://conda.anaconda.org/conda-forge/win-64/libsvm-331-h63175ca_0.conda
https://repo.anaconda.com/pkgs/main/win-64/libuv-1.44.2-h2bbff1b_0.conda
https://conda.anaconda.org/conda-forge/win-64/libwebp-base-1.3.1-hcfcfb64_0.conda
https://conda.anaconda.org/conda-forge/win-64/libzlib-1.2.13-hcfcfb64_5.conda
https://conda.anaconda.org/conda-forge/win-64/lz4-c-1.9.4-hcfcfb64_0.conda
https://conda.anaconda.org/conda-forge/win-64/lzo-2.10-he774522_1000.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/m2w64-gcc-libs-5.3.0-7.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/openssl-3.1.1-hcfcfb64_1.conda
https://conda.anaconda.org/conda-forge/win-64/pixman-0.40.0-h8ffe710_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pthreads-win32-2.9.1-hfa6e2cd_3.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/reproc-14.2.4-hcfcfb64_0.conda
https://conda.anaconda.org/conda-forge/win-64/snappy-1.1.10-hfb803bf_0.conda
https://repo.anaconda.com/pkgs/main/win-64/sqlite-3.41.2-h2bbff1b_0.conda
https://conda.anaconda.org/conda-forge/win-64/tk-8.6.12-h8ffe710_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/xz-5.2.6-h8d14728_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/yaml-0.2.5-h8ffe710_2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/yaml-cpp-0.7.0-h63175ca_2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/gettext-0.21.1-h5728263_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/hdf4-4.2.15-h1334946_6.conda
https://conda.anaconda.org/conda-forge/win-64/krb5-1.20.1-heb0366b_0.conda
https://conda.anaconda.org/conda-forge/win-64/libbrotlidec-1.0.9-hcfcfb64_9.conda
https://conda.anaconda.org/conda-forge/win-64/libbrotlienc-1.0.9-hcfcfb64_9.conda
https://conda.anaconda.org/conda-forge/win-64/libclang13-15.0.7-default_h77d9078_2.conda
https://conda.anaconda.org/conda-forge/win-64/libpng-1.6.39-h19919ed_0.conda
https://conda.anaconda.org/conda-forge/win-64/libsolv-0.7.24-h12be248_1.conda
https://conda.anaconda.org/conda-forge/win-64/libssh2-1.11.0-h7dfc565_0.conda
https://conda.anaconda.org/conda-forge/win-64/libvorbis-1.3.7-h0e60522_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libxml2-2.11.4-hc3477c8_0.conda
https://conda.anaconda.org/conda-forge/win-64/libzip-1.9.2-h519de47_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pcre2-10.40-h17e33f8_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pthread-stubs-0.4-hcd874cb_1001.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/python-3.9.7-h900ac77_3_cpython.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/reproc-cpp-14.2.4-h63175ca_0.conda
https://conda.anaconda.org/conda-forge/win-64/xorg-libxau-1.0.11-hcd874cb_0.conda
https://conda.anaconda.org/conda-forge/win-64/xorg-libxdmcp-1.1.3-hcd874cb_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/zeromq-4.3.4-h0e60522_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/zlib-1.2.13-hcfcfb64_5.conda
https://conda.anaconda.org/conda-forge/win-64/zstd-1.5.2-h12be248_6.conda
https://conda.anaconda.org/conda-forge/win-64/blosc-1.21.4-hdccc3a2_0.conda
https://conda.anaconda.org/conda-forge/win-64/boost-cpp-1.78.0-h9f4b32c_3.conda
https://conda.anaconda.org/conda-forge/win-64/brotli-bin-1.0.9-hcfcfb64_9.conda
https://conda.anaconda.org/conda-forge/win-64/freetype-2.12.1-h546665d_1.conda
https://conda.anaconda.org/conda-forge/win-64/libarchive-3.6.2-h6f8411a_1.conda
https://conda.anaconda.org/conda-forge/win-64/libclang-15.0.7-default_h77d9078_2.conda
https://conda.anaconda.org/conda-forge/win-64/libcurl-8.1.2-h68f0423_0.conda
https://conda.anaconda.org/conda-forge/win-64/libglib-2.76.3-he8f3873_0.conda
https://conda.anaconda.org/conda-forge/win-64/libhwloc-2.9.1-nocuda_h15da153_6.conda
https://conda.anaconda.org/conda-forge/win-64/libtiff-4.5.1-h6c8260b_0.conda
https://conda.anaconda.org/conda-forge/win-64/libxcb-1.15-hcd874cb_0.conda
https://conda.anaconda.org/conda-forge/win-64/python_abi-3.9-2_cp39.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/brotli-1.0.9-hcfcfb64_9.conda
https://conda.anaconda.org/conda-forge/win-64/curl-8.1.2-h68f0423_0.conda
https://conda.anaconda.org/conda-forge/win-64/fontconfig-2.14.2-hbde0cde_0.conda
https://conda.anaconda.org/conda-forge/win-64/glib-tools-2.76.3-h12be248_0.conda
https://conda.anaconda.org/conda-forge/win-64/hdf5-1.14.1-nompi_h73e8ff5_100.conda
https://conda.anaconda.org/conda-forge/win-64/lcms2-2.15-h3e3b177_1.conda
https://conda.anaconda.org/conda-forge/win-64/libmamba-1.4.6-ha24f096_0.conda
https://conda.anaconda.org/conda-forge/win-64/menuinst-1.4.19-py39hcbf5309_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/openjpeg-2.5.0-ha2aaf27_2.conda
https://conda.anaconda.org/conda-forge/win-64/tbb-2021.9.0-h91493d7_0.conda
https://conda.anaconda.org/conda-forge/noarch/boltons-23.0.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/brotli-python-1.0.9-py39h99910a6_9.conda
https://conda.anaconda.org/conda-forge/win-64/cairo-1.16.0-hdecc03f_1016.conda
https://repo.anaconda.com/pkgs/main/win-64/cython-0.29.35-py39hd77b12b_0.conda
https://conda.anaconda.org/conda-forge/win-64/debugpy-1.6.7-py39h99910a6_0.conda
https://repo.anaconda.com/pkgs/main/win-64/filelock-3.9.0-py39haa95532_0.conda
https://conda.anaconda.org/conda-forge/win-64/glib-2.76.3-h12be248_0.conda
https://conda.anaconda.org/conda-forge/win-64/kiwisolver-1.4.4-py39h1f6ef14_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libmambapy-1.4.6-py39h608c70e_0.conda
https://conda.anaconda.org/conda-forge/win-64/libnetcdf-4.9.2-nompi_h5729074_106.conda
https://conda.anaconda.org/conda-forge/win-64/markupsafe-2.1.3-py39ha55989b_0.conda
https://conda.anaconda.org/conda-forge/win-64/mkl-2022.1.0-h6a75c08_874.tar.bz2
https://conda.anaconda.org/salilab/win-64/modeller-10.4-py39he774522_0.tar.bz2
https://repo.anaconda.com/pkgs/main/win-64/mpmath-1.2.1-py39haa95532_0.conda
https://conda.anaconda.org/conda-forge/win-64/msgpack-python-1.0.5-py39h1f6ef14_0.conda
https://conda.anaconda.org/conda-forge/noarch/packaging-23.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/pillow-10.0.0-py39ha9166d5_0.conda
https://conda.anaconda.org/conda-forge/noarch/pluggy-1.2.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/propka-3.5.0-py39h00e5391_0.conda
https://conda.anaconda.org/conda-forge/win-64/psutil-5.9.5-py39ha55989b_0.conda
https://conda.anaconda.org/conda-forge/win-64/pycosat-0.6.4-py39ha55989b_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pyrsistent-0.19.3-py39ha55989b_0.conda
https://conda.anaconda.org/conda-forge/win-64/pywin32-304-py39h99910a6_2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pywinpty-2.0.10-py39h99910a6_0.conda
https://conda.anaconda.org/conda-forge/win-64/pyyaml-6.0-py39ha55989b_5.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pyzmq-25.1.0-py39hea35a22_0.conda
https://conda.anaconda.org/conda-forge/win-64/ruamel.yaml.clib-0.2.7-py39ha55989b_1.conda
https://repo.anaconda.com/pkgs/main/win-64/setuptools-67.8.0-py39haa95532_0.conda
https://repo.anaconda.com/pkgs/main/win-64/tenacity-8.2.2-py39haa95532_0.conda
https://conda.anaconda.org/conda-forge/noarch/toolz-0.12.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/tornado-6.3.2-py39ha55989b_0.conda
https://conda.anaconda.org/conda-forge/win-64/unicodedata2-15.0.0-py39ha55989b_0.tar.bz2
https://repo.anaconda.com/pkgs/main/win-64/wheel-0.38.4-py39haa95532_0.conda
https://conda.anaconda.org/conda-forge/win-64/gstreamer-1.22.4-hb4038d2_0.conda
https://conda.anaconda.org/conda-forge/win-64/libblas-3.9.0-17_win64_mkl.conda
https://repo.anaconda.com/pkgs/main/win-64/pip-23.1.2-py39haa95532_0.conda
https://conda.anaconda.org/conda-forge/win-64/pycairo-1.24.0-py39h82915f6_0.conda
https://conda.anaconda.org/conda-forge/win-64/ruamel.yaml-0.17.32-py39ha55989b_0.conda
https://repo.anaconda.com/pkgs/main/win-64/sympy-1.11.1-py39haa95532_0.conda
https://conda.anaconda.org/conda-forge/win-64/gst-plugins-base-1.22.4-h001b923_0.conda
https://conda.anaconda.org/conda-forge/win-64/libcblas-3.9.0-17_win64_mkl.conda
https://conda.anaconda.org/conda-forge/win-64/liblapack-3.9.0-17_win64_mkl.conda
https://conda.anaconda.org/conda-forge/win-64/numpy-1.25.0-py39h816b6a6_0.conda
https://conda.anaconda.org/conda-forge/win-64/qt-main-5.15.8-h2c8576c_14.conda
https://conda.anaconda.org/conda-forge/win-64/biopython-1.81-py39ha55989b_0.conda
https://conda.anaconda.org/conda-forge/win-64/boost-1.78.0-py39hbd792c9_4.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/cftime-1.6.2-py39hc266a54_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/contourpy-1.1.0-py39h1f6ef14_0.conda
https://conda.anaconda.org/conda-forge/win-64/gsd-2.9.0-py39hbaa61f9_0.conda
https://conda.anaconda.org/conda-forge/win-64/openmm-8.0.0-py39h6259327_1.conda
https://conda.anaconda.org/conda-forge/win-64/pytng-0.3.0-py39hcb05468_0.conda
https://conda.anaconda.org/conda-forge/noarch/attrs-23.1.0-pyh71513ae_1.conda
https://conda.anaconda.org/conda-forge/noarch/backcall-0.2.0-pyh9f0ad1d_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/backports-1.0-pyhd8ed1ab_3.conda
https://conda.anaconda.org/conda-forge/noarch/backports.functools_lru_cache-1.6.5-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/cached_property-1.5.2-pyha770c72_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/cached-property-1.5.2-hd8ed1ab_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/h5py-3.9.0-nompi_py39hde1a6dc_101.conda
https://conda.anaconda.org/conda-forge/noarch/certifi-2023.5.7-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/netcdf4-1.6.4-nompi_py39h6379b8f_101.conda
https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-3.1.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.6-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/tqdm-4.65.0-pyhd8ed1ab_1.conda
https://conda.anaconda.org/conda-forge/noarch/conda-package-handling-2.0.2-pyh38be061_0.conda
https://conda.anaconda.org/conda-forge/noarch/cycler-0.11.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/decorator-5.1.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/defusedxml-0.7.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/entrypoints-0.4-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/exceptiongroup-1.1.2-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/executing-1.2.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/fasteners-0.17.3-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/flit-core-3.9.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/freetype-py-2.3.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/future-0.18.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/idna-3.4-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/importlib_metadata-6.7.0-hd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/ipython_genutils-0.2.0-py_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/jinja2-3.1.2-pyhd8ed1ab_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/joblib-1.3.0-pyhd8ed1ab_1.conda
https://conda.anaconda.org/conda-forge/noarch/jsonpatch-1.32-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/jsonpointer-2.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/jupyterlab_widgets-1.1.4-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/mistune-3.0.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/mmtf-python-1.1.3-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/mrcfile-1.4.3-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/munkres-1.1.4-pyh9f0ad1d_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/fonttools-4.40.0-py39ha55989b_0.conda
https://conda.anaconda.org/conda-forge/noarch/nest-asyncio-1.5.6-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/networkx-3.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pandocfilters-1.5.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/parso-0.8.3-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/jedi-0.18.2-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pbr-5.11.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pickleshare-0.7.5-py_1003.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pkgutil-resolve-name-1.3.10-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/ply-3.11-py_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/prometheus_client-0.17.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/prompt_toolkit-3.0.38-hd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pure_eval-0.2.2-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pycparser-2.21-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/cffi-1.15.1-py39h68f70e3_3.conda
https://conda.anaconda.org/conda-forge/win-64/argon2-cffi-bindings-21.2.0-py39ha55989b_3.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/cryptography-41.0.1-py39hb6bd5e6_0.conda
https://conda.anaconda.org/conda-forge/win-64/zstandard-0.19.0-py39h95af829_2.conda
https://conda.anaconda.org/conda-forge/noarch/pyopenssl-23.2.0-pyhd8ed1ab_1.conda
https://conda.anaconda.org/conda-forge/noarch/argon2-cffi-21.3.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/conda-package-streaming-0.8.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pyedr-0.7.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pygments-2.15.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/jupyterlab_pygments-0.2.2-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pyparsing-3.1.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/python-fastjsonschema-2.17.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/python-json-logger-2.0.7-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/python-tzdata-2023.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pytz-2023.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/qtpy-2.3.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/requests-2.31.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/conda-23.5.0-py39hcbf5309_1.conda
https://conda.anaconda.org/conda-forge/win-64/mamba-1.4.6-py39hca8391b_0.conda
https://conda.anaconda.org/plotly/noarch/plotly-5.15.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/rfc3986-validator-0.1.1-pyh9f0ad1d_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/rlpycairo-0.2.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/send2trash-1.8.2-pyh08f2357_0.conda
https://conda.anaconda.org/conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.3.2.post1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/terminado-0.17.0-pyh08f2357_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/threadpoolctl-3.1.0-pyh8a188c0_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/tidynamics-1.0.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/toml-0.10.2-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/tomli-2.0.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/traitlets-5.9.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.7.1-pyha770c72_0.conda
https://conda.anaconda.org/conda-forge/noarch/typing_utils-0.1.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/wcwidth-0.2.6-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/webencodings-0.5.1-py_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/websocket-client-1.6.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/win_inet_pton-1.1.0-pyhd8ed1ab_6.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/zipp-3.15.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/anyio-3.7.0-pyhd8ed1ab_1.conda
https://conda.anaconda.org/conda-forge/noarch/asttokens-2.2.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/beautifulsoup4-4.12.2-pyha770c72_0.conda
https://conda.anaconda.org/conda-forge/noarch/bleach-6.0.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/comm-0.1.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-6.7.0-pyha770c72_0.conda
https://conda.anaconda.org/conda-forge/noarch/importlib_resources-5.12.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/jupyter_server_terminals-0.4.4-pyhd8ed1ab_1.conda
https://conda.anaconda.org/conda-forge/noarch/matplotlib-inline-0.1.6-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/overrides-7.3.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/prompt-toolkit-3.0.38-pyha770c72_0.conda
https://conda.anaconda.org/conda-forge/noarch/pysocks-1.7.1-pyh0701188_6.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.8.2-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/pytorch/win-64/pytorch-2.0.1-py3.9_cpu_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/reportlab-4.0.4-py39ha55989b_0.conda
https://conda.anaconda.org/conda-forge/noarch/rfc3339-validator-0.1.4-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/sip-6.7.9-py39h99910a6_0.conda
https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.2.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/typing-extensions-4.7.1-hd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/importlib-resources-5.12.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/jsonschema-4.17.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/pandas-2.0.3-py39h1679cfb_0.conda
https://conda.anaconda.org/conda-forge/noarch/platformdirs-3.8.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/pyqt5-sip-12.11.0-py39h99910a6_3.conda
https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.2-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/urllib3-2.0.3-pyhd8ed1ab_1.conda
https://conda.anaconda.org/conda-forge/noarch/ipython-8.14.0-pyh08f2357_0.conda
https://conda.anaconda.org/conda-forge/win-64/jupyter_core-5.3.1-py39hcbf5309_0.conda
https://conda.anaconda.org/conda-forge/noarch/jupyter_events-0.6.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/matplotlib-base-3.7.1-py39haf65ace_0.conda
https://conda.anaconda.org/conda-forge/noarch/pooch-1.7.0-pyha770c72_3.conda
https://conda.anaconda.org/conda-forge/win-64/pyqt-5.15.7-py39hb77abff_3.conda
https://conda.anaconda.org/conda-forge/noarch/jupyter_client-8.3.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/nbformat-5.9.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/scipy-1.11.1-py39hde5eda1_0.conda
https://conda.anaconda.org/conda-forge/win-64/cctbx-base-2023.6-py39ha880560_0.conda
https://conda.anaconda.org/conda-forge/noarch/griddataformats-1.0.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/ipykernel-6.24.0-pyh6817e22_0.conda
https://conda.anaconda.org/conda-forge/noarch/nbclient-0.8.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/patsy-0.5.3-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/scikit-learn-1.3.0-py39hfa9d973_0.conda
https://conda.anaconda.org/conda-forge/noarch/seaborn-base-0.12.2-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/biobox-1.1.1-py39hc266a54_2.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/jupyter_console-6.6.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/nbconvert-core-7.6.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/qtconsole-base-5.4.3-pyha770c72_0.conda
https://conda.anaconda.org/conda-forge/win-64/statsmodels-0.14.0-py39hbaa61f9_1.conda
https://conda.anaconda.org/conda-forge/noarch/jupyter_server-2.7.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/nbconvert-pandoc-7.6.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/qtconsole-5.4.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/seaborn-0.12.2-hd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/win-64/mdanalysis-2.5.0-py39h614b579_1.conda
https://conda.anaconda.org/conda-forge/noarch/nbconvert-7.6.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/notebook-shim-0.2.3-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/nbclassic-1.0.0-pyhb4ecaf3_1.conda
https://conda.anaconda.org/conda-forge/noarch/notebook-6.5.4-pyha770c72_0.conda
https://conda.anaconda.org/conda-forge/noarch/widgetsnbextension-3.6.4-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/ipywidgets-7.7.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/jupyter-1.0.0-py39hcbf5309_8.conda
https://conda.anaconda.org/conda-forge/noarch/nglview-3.0.6-pyhba93850_0.conda
RMeli commented 1 year ago

One thing I would like to verify, is whether this crash still takes place if you reduce the number of parallel processes running, i.e., setting MA.processes = 2 in In[7].

Are you talking about the molearn_GUI notebook or the molearn_analysis notebook? I'm talking about the latter (the former works great now besides the visualization hiccups, sorry for the confusion). The crash happens in the "RMSD (or MSE) of input vs autoencoded".

Unfortunately I don't have access to a Windows machine so I can't fully reproduce your environment, but I'll try to pin the version of the main dependencies to the same, thanks.

I am investigating the problem with nglview. I am currently running on a new laptop with a fresh environment, and I can confirm that (while calculations above run as intended), the 3D protein visualisation and traces on the latent space are not visible.

Interesting, thanks for checking.

SCMusson commented 1 year ago

The memory problem at In[12] and In[17] seems to be due to the code attempting to encode/decode the entire dataset at once. I've added a batched encode and decode methods that makes the job far more reasonable. I believe the datasets provided in the data directory have been enlarged since molearn_analysis was originally written and the encoder has been changed to a graph neural network which is far more memory intensive than a convolutional net.

molearn_GUI uses molearn.analysis.MolearnAnalysis which automatically batches the calculations anyway hence why we haven't discovered this before.

The notebook should now run however @degiacom may want to modify the text or present the changes differently.

RMeli commented 1 year ago

@SCMusson thanks for checking and providing a fix. I can confirm that with batching everything works well! I haven't seen RAM usage go past 10%.

(And thanks for adding progress bars, they are great from a first-time user perspective!)

Note: I only tested up to "EXTRA MATERIAL (work in progress)".

degiacom commented 1 year ago

Thank you @SCMusson. In molearn_analysis, I have only slightly edited the textual explanations, and removed the "EXTRA MATERIAL (work in process)" section for clarity.

RMeli commented 1 year ago

Thanks for updating the notebooks, LGTM! I think the only outstanding issue here is the NGLview one.

degiacom commented 1 year ago

@RMeli, we seem to have identified the problem with nglview. This appears linked to a know issue with how Jupyter notebook interacts with asynchronous calls. A currently known workaround is to downgrade tornado to 6.1. In our hands, this fix works with both Linux and Windows. We have updated the FAQ, reporting on the combination of packages that will work. There is also a minor debug to the GUI itself.

RMeli commented 1 year ago

Fantastic @degiacom! I had indeed tornado==6.3.2 installed, and I can confirm that downgrading it to 6.1 does the trick: path selection works as expected and I can see the 3D renders!

Thanks for tracking this down. It will be worth to pin this dependency in the next conda-forge release, but given the updated FAQ looks good to me!