Issue with Typing-extensions during run of batch submission.

[coopersvajda]

Hi there, So I have been battling to get ESM on my local university cluster for a few days now. I have managed to get all of the issues related to nvcc/torch/cuda compatibility to download openfold (after much struggling). But now, when I run a test submission:

python ./scripts/esmfold_inference.py -i fasta.fasta -o ./ --cpu-only

It gives the following set of error messages:

23/02/16 13:03:46 | INFO | root | Reading sequences from fasta.fasta 23/02/16 13:03:46 | INFO | root | Loaded 1 sequences from fasta.fasta 23/02/16 13:03:46 | INFO | root | Loading model Traceback (most recent call last): File "/scratch/user/cooper1995/dream/ESM/./scripts/esmfold_inference.py", line 136, in model = esm.pretrained.esmfold_v1() File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/esm/pretrained.py", line 419, in esmfold_v1 import esm.esmfold.v1.pretrained File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/esm/esmfold/v1/pretrained.py", line 5, in from esm.esmfold.v1.esmfold import ESMFold File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/esm/esmfold/v1/esmfold.py", line 11, in from openfold.data.data_transforms import make_atom14_masks File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/init.py", line 1, in from . import model File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/model/init.py", line 11, in _modules = [(m, importlib.import_module("." + m, name)) for m in all] File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/model/init.py", line 11, in _modules = [(m, importlib.import_module("." + m, name)) for m in all] File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/model/msa.py", line 21, in from openfold.model.primitives import ( File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/model/primitives.py", line 35, in from openfold.utils.checkpointing import get_checkpoint_fn File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/utils/init.py", line 13, in _modules = [(m, importlib.import_module("." + m, name)) for m in all] File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/utils/init.py", line 13, in _modules = [(m, importlib.import_module("." + m, name)) for m in all] File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/openfold/utils/logger.py", line 21, in from pytorch_lightning import Callback File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/pytorch_lightning/init.py", line 34, in from lightning_fabric.utilities.seed import seed_everything # noqa: E402 File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/lightning_fabric/init.py", line 23, in from lightning_fabric.fabric import Fabric # noqa: E402 File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/lightning_fabric/fabric.py", line 34, in from lightning_fabric.connector import _Connector, _PLUGIN_INPUT, _PRECISION_INPUT File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/lightning_fabric/connector.py", line 45, in from lightning_fabric.strategies import ( File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/lightning_fabric/strategies/init.py", line 14, in from lightning_fabric.strategies.ddp import DDPStrategy # noqa: F401 File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/lightning_fabric/strategies/ddp.py", line 26, in from lightning_fabric.plugins.collectives.torch_collective import default_pg_timeout File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/lightning_fabric/plugins/collectives/init.py", line 1, in from lightning_fabric.plugins.collectives.collective import Collective File "/scratch/user/cooper1995/.conda/envs/SE3-nvidia/lib/python3.9/site-packages/lightning_fabric/plugins/collectives/collective.py", line 5, in from typing_extensions import Self ImportError: cannot import name 'Self' from 'typing_extensions' (/sw/eb/sw/typing-extensions/3.10.0.0-GCCcore-10.3.0/lib/python3.9/site-packages/typing_extensions.py)

My environment, and the contained packages are as follows:

**# Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge absl-py 1.3.0 py39h06a4308_0 aiohttp 3.8.4 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi antlr4-python3-runtime 4.9.3 pypi_0 pypi anyio 3.3.0 py39hf3d152e_0 conda-forge argon2-cffi 20.1.0 py39h3811e60_2 conda-forge asttokens 2.2.1 pyhd8ed1ab_0 conda-forge astunparse 1.6.3 pypi_0 pypi async-timeout 4.0.2 pypi_0 pypi async_generator 1.10 py_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge babel 2.9.1 pyh44b312d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.11.1 py39h06a4308_0 biopython 1.78 py39h7f8727e_0 blas 1.0 mkl conda-forge bleach 4.1.0 pyhd8ed1ab_0 conda-forge brotlipy 0.7.0 py39h3811e60_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge ca-certificates 2022.12.7 ha878542_0 conda-forge cachetools 5.2.0 pypi_0 pypi certifi 2022.12.7 pyhd8ed1ab_0 conda-forge cffi 1.14.6 py39he32792d_0 conda-forge chardet 4.0.0 py39hf3d152e_1 conda-forge charset-normalizer 2.0.0 pyhd8ed1ab_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge contextlib2 21.6.0 pypi_0 pypi cryptography 3.4.7 py39hbca0aa6_0 conda-forge cudatoolkit 11.1.74 h6bb024c_0 nvidia cycler 0.11.0 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h48d8840_2 conda-forge debugpy 1.4.1 py39he80948d_0 conda-forge decorator 5.0.9 pyhd8ed1ab_0 conda-forge deepspeed 0.5.9 pypi_0 pypi defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge dgl-cuda11.1 0.8.2 py39_0 dglteam dllogger 1.0.0 pypi_0 pypi dm-haiku 0.0.8 pypi_0 pypi dm-tree 0.1.5 py39h51133e4_0 e3nn 0.3.4 pypi_0 pypi einops 0.6.0 pypi_0 pypi entrypoints 0.3 pyhd8ed1ab_1003 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge expat 2.4.1 h9c3ff4c_0 conda-forge fair-esm 2.0.0 pypi_0 pypi ffmpeg 4.3 hf484d3e_0 pytorch fftw 3.3.9 h27cfd23_1 flatbuffers 22.10.26 pypi_0 pypi fontconfig 2.13.1 he4413a7_1000 conda-forge freetype 2.10.4 h0708190_1 conda-forge frozenlist 1.3.3 pypi_0 pypi fsspec 2023.1.0 pypi_0 pypi gast 0.4.0 pypi_0 pypi gemmi 0.4.8 pypi_0 pypi gettext 0.19.8.1 h0b5b191_1005 conda-forge glib 2.68.4 h9c3ff4c_0 conda-forge glib-tools 2.68.4 h9c3ff4c_0 conda-forge gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge google-auth 2.14.1 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi googledrivedownloader 0.4 pyhd3deb0d_1 conda-forge grpcio 1.50.0 pypi_0 pypi gst-plugins-base 1.14.0 hbbd80ab_1 gstreamer 1.14.0 h28cd5cc_2 h5py 3.7.0 pypi_0 pypi hjson 3.1.0 pypi_0 pypi icecream 2.1.3 pyhd8ed1ab_0 conda-forge icu 58.2 hf484d3e_1000 conda-forge idna 3.1 pyhd3deb0d_0 conda-forge importlib-metadata 4.8.1 py39hf3d152e_0 conda-forge intel-openmp 2021.3.0 h06a4308_3350 ipykernel 6.13.0 py39hef51801_0 conda-forge ipython 7.27.0 py39hef51801_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.18.0 py39hf3d152e_2 conda-forge jinja2 3.0.1 pyhd8ed1ab_0 conda-forge jmp 0.0.2 pypi_0 pypi joblib 1.0.1 pyhd8ed1ab_0 conda-forge jpeg 9b h024ee3a_2 json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge jupyter_client 7.0.2 pyhd8ed1ab_0 conda-forge jupyter_core 4.7.1 py39hf3d152e_0 conda-forge jupyter_server 1.11.0 pyhd8ed1ab_0 conda-forge jupyterlab 3.1.12 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 2.8.1 pyhd8ed1ab_0 conda-forge keras 2.10.0 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.3.2 py39h1a9c180_1 conda-forge lame 3.100 h7f98852_1001 conda-forge lcms2 2.12 h3be6417_0 ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge libclang 14.0.6 pypi_0 pypi libffi 3.3 h58526e2_2 conda-forge libgcc-ng 11.1.0 hc902ee8_8 conda-forge libgfortran-ng 7.5.0 h14aa051_19 conda-forge libgfortran4 7.5.0 h14aa051_19 conda-forge libglib 2.68.4 h3e27bee_0 conda-forge libgomp 11.1.0 hc902ee8_8 conda-forge libiconv 1.16 h516909a_0 conda-forge libllvm11 11.1.0 hf817b99_2 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libstdcxx-ng 11.1.0 h56837e0_8 conda-forge libtiff 4.2.0 h85742a9_0 libuuid 2.32.1 h7f98852_1000 conda-forge libuv 1.42.0 h7f98852_0 conda-forge libwebp-base 1.2.1 h7f98852_0 conda-forge libxcb 1.13 h7f98852_1003 conda-forge libxml2 2.9.9 h13577e0_2 conda-forge lie-learn 0.0.1.post1 pypi_0 pypi lightning-utilities 0.6.0.post0 pypi_0 pypi llvmlite 0.37.0 py39h1bbdace_0 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge markdown 3.4.1 pypi_0 pypi markupsafe 2.1.1 py39hb9d737c_1 conda-forge matplotlib 3.4.3 py39hf3d152e_1 conda-forge matplotlib-base 3.4.3 py39h2fa2bec_1 conda-forge matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge mistune 0.8.4 py39h3811e60_1004 conda-forge mkl 2021.3.0 h06a4308_520 mkl-service 2.4.0 py39h3811e60_0 conda-forge mkl_fft 1.3.0 py39h42c9631_2 mkl_random 1.2.2 py39hde0f152_0 conda-forge ml-collections 0.1.1 pypi_0 pypi mpmath 1.2.1 pypi_0 pypi multidict 6.0.4 pypi_0 pypi nbclassic 0.3.1 pyhd8ed1ab_1 conda-forge nbclient 0.5.4 pyhd8ed1ab_0 conda-forge nbconvert 6.4.4 py39h06a4308_0 nbformat 5.1.3 pyhd8ed1ab_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge nettle 3.6 he412f7d_0 conda-forge networkx 2.5 py_0 conda-forge ninja 1.11.1 pypi_0 pypi notebook 6.4.4 pyha770c72_0 conda-forge numba 0.54.0 py39h56b8d98_0 conda-forge numpy 1.20.3 py39hf144106_0 numpy-base 1.20.3 py39h74d4b33_0 nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi ocl-icd 2.3.1 h7f98852_0 conda-forge ocl-icd-system 1.0.0 1 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge omegaconf 2.3.0 pypi_0 pypi openfold 1.0.0 pypi_0 pypi openh264 2.1.1 h780b84a_0 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openmm 7.6.0 py39h33ab577_0 conda-forge openssl 1.1.1l h7f98852_0 conda-forge opt-einsum 3.3.0 pypi_0 pypi opt-einsum-fx 0.1.2 pypi_0 pypi packaging 21.0 pyhd8ed1ab_0 conda-forge pandas 1.3.2 py39hde0f152_0 conda-forge pandoc 2.14.2 h7f98852_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge patsy 0.5.2 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pdbfixer 1.8.1 pyh6c4a22f_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.3.1 py39h2c7a002_0 pip 21.2.4 pyhd8ed1ab_0 conda-forge prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.20 pyha770c72_0 conda-forge protobuf 3.19.6 pypi_0 pypi psutil 5.9.0 py39hb9d737c_1 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge py-cpuinfo 9.0.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.10.0 pyhd8ed1ab_0 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyqt 5.9.2 py39h2531618_6 pyrsistent 0.17.3 py39h3811e60_2 conda-forge pysocks 1.7.1 py39hf3d152e_3 conda-forge python 3.9.6 h49503c6_1_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-louvain 0.15 pyhd3deb0d_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytorch-cluster 1.5.9 py39_torch_1.9.0_cu111 rusty1s pytorch-geometric 1.7.2 py39_torch_1.9.0_cu111 rusty1s pytorch-lightning 1.9.2 pypi_0 pypi pytorch-scatter 2.0.8 py39_torch_1.9.0_cu111 rusty1s pytorch-sparse 0.6.11 py39_torch_1.9.0_cu111 rusty1s pytorch-spline-conv 1.2.1 py39_torch_1.9.0_cu111 rusty1s pytz 2021.1 pyhd8ed1ab_0 conda-forge pyyaml 6.0 pypi_0 pypi pyzmq 22.2.1 py39h37b5a0c_0 conda-forge qt 5.9.7 h5867ecd_1 readline 8.1 h46c0cb4_0 conda-forge requests 2.26.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pypi_0 pypi requests-unixsocket 0.2.0 py_0 conda-forge rsa 4.9 pypi_0 pypi scikit-learn 0.24.2 py39ha9443f7_0 scipy 1.6.2 py39had2a1c9_1 seaborn 0.11.2 hd8ed1ab_0 conda-forge seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 57.4.0 py39hf3d152e_0 conda-forge sip 4.19.13 py39h2531618_0 six 1.16.0 pyh6c4a22f_0 conda-forge sniffio 1.2.0 py39hf3d152e_1 conda-forge soupsieve 2.3.1 pyhd3eb1b0_0 sqlite 3.36.0 h9cd32fc_0 conda-forge statsmodels 0.13.2 py39hce5d2b2_0 conda-forge sympy 1.8 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tensorboard 2.10.1 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow 2.10.0 pypi_0 pypi tensorflow-estimator 2.10.0 pypi_0 pypi tensorflow-io-gcs-filesystem 0.27.0 pypi_0 pypi termcolor 2.1.0 pypi_0 pypi terminado 0.12.1 py39hf3d152e_0 conda-forge testpath 0.5.0 pyhd8ed1ab_0 conda-forge threadpoolctl 2.2.0 pyh8a188c0_0 conda-forge tk 8.6.11 h21135ba_0 conda-forge torch 1.13.1 pypi_0 pypi torchaudio 0.9.0 py39 pytorch torchmetrics 0.11.1 pypi_0 pypi torchvision 0.10.0 py39_cu111 pytorch tornado 6.1 py39h3811e60_1 conda-forge tqdm 4.62.2 pyhd8ed1ab_0 conda-forge traitlets 5.1.0 pyhd8ed1ab_0 conda-forge triton 1.0.0 pypi_0 pypi typing-extensions 4.5.0 pypi_0 pypi tzdata 2021a he74cb21_1 conda-forge urllib3 1.26.6 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 0.57.0 py39hf3d152e_4 conda-forge werkzeug 2.2.2 pypi_0 pypi wheel 0.37.0 pyhd8ed1ab_1 conda-forge wrapt 1.14.1 pypi_0 pypi xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.5 h516909a_1 conda-forge yarl 1.8.2 pypi_0 pypi zeromq 4.3.4 h9c3ff4c_1 conda-forge zipp 3.5.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.4.9 ha95c52a_0 conda-forge**

This environment is pretty beefy, but it is because it is a shared environment for two other ML programs also pertaining to proteins. The most glaring potential issue is the fact that I have typing-extensions 4.5.0 instead of 3.10.0.2 as described in the environment.yml file. Is this an issue that could be resolved by downgrading my version of typing-extensions? Is there a workaround to this problem/is this a problem others have seen?

Thank for any help in this regard! Cooper

[nikitos9000]

Hi, could you please try using a separate environment built with our environment.yml file?

[JasperGeissler]

Hello,

May I ask how you managed to get openfold to load properly? I've run into a fair share of issues trying to get it to cooperate.

Thanks in advance!

[coopersvajda]

Hi there, I actually went ahead and asked the HPRC staff at my University to install ESMFold as a module on our local supercomputing cluster. I'm sorry I do not have a solution for you.

Cooper Svajda Research Assistant/PhD Student Plant Pathology and Microbiology Department Texas A&M University

---

From: JasperGeissler @.> Sent: Monday, March 27, 2023 10:21 AM To: facebookresearch/esm @.> Cc: Cooper Svajda @.>; Author @.> Subject: Re: [facebookresearch/esm] Issue with Typing-extensions during run of batch submission. (Issue #481)

Hello,

May I ask how you managed to get openfold to load properly? I've run into a fair share of issues trying to get it to cooperate.

Thanks in advance!

— Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fesm%2Fissues%2F481%23issuecomment-1485326453&data=05%7C01%7Ccooper.svajda%40ag.tamu.edu%7C228d771683124a67bbe208db2ed6e0d7%7C9fd7580a64724d9ca142d131d3a7a116%7C0%7C0%7C638155272654499749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mPcYt7jQ8icloCuLksSRkjVZFI%2FIrQU9nc2pT%2FZqC64%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA2AKQ6WUMUXG24GHVKAXKR3W6GV55ANCNFSM6AAAAAAU6R7I3E&data=05%7C01%7Ccooper.svajda%40ag.tamu.edu%7C228d771683124a67bbe208db2ed6e0d7%7C9fd7580a64724d9ca142d131d3a7a116%7C0%7C0%7C638155272654499749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fFIJig10HZ3uI0N98OH30ZOyKkOaVHeiEQJYEqEYg%2BM%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

[tomsercu]

@JasperGeissler you can open a separate issue with a precise description of the errors? The OP message looks like something is wrong in the base python installation, unlikely this would be the same issue.. I also want to mention that the huggingface transformer port won't run into this dependency tangle, if that's an option for you.