search

rsanchezgarc / deepEMhancer

Deep learning for cryo-EM maps post-processing
Apache License 2.0
49 stars 8 forks source link

Install error GLIBCXX_3.4.26 not found #25

Closed sakoht closed 1 year ago

sakoht commented 1 year ago

I'm attempting to install deepEMhancer and running into a problem on any base Linux OS I've tried.

Is there a preferred Linux OS/version for which the install is known to work? (I'm running in VMs so can pick whatever!)

Advice on making it work from scratch on Ubuntu 20.04, 22.04? Or whatever older OS it was developed on is great too!!

Thanks in advance.

rsanchezgarc commented 1 year ago

Hi,

I am using Ubuntu 20, and I never had those problems. What installation recipe are you following? It looks like something like numpy was installed using conda, and then, pip changed it so that the library does not match.

I am also curious abut ubuntu 22 not being compatible with python 3.9. Conda should be able to manage it. Can you please paste here the error logs you get in any case, so that I get a better understanding?

rjgildea commented 1 year ago

It seems that pip install tensorflow=2.12.0 triggers a downgrade of numpy from that installed by conda which results in the error observed. I've managed to reproduce the issue in a standalone mamba environment:

$ mamba create -n tmp python=3.9 scipy=1.9 -y # from deepEMhancer_env.yml
$ mamba activate tmp
$ mamba list | grep numpy
numpy                     1.25.2           py39h6183b62_0    conda-forge
$ mamba list | grep scipy
scipy                     1.9.3            py39hddc5342_2    conda-forge
$ python -c "import scipy.linalg" # OK
$ pip install tensorflow==2.12.0 # from https://github.com/rsanchezgarc/deepEMhancer/blob/80e16b112df62a7e99b3e196353bdd8789ed3ab4/deepEMhancer_env.yml#L46
...
Installing collected packages: libclang, flatbuffers, zipp, wrapt, urllib3, typing-extensions, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, pyasn1, protobuf, packaging, oauthlib, numpy, MarkupSafe, keras, idna, grpcio, gast, charset-normalizer, certifi, cachetools, absl-py, werkzeug, rsa, requests, pyasn1-modules, opt-einsum, ml-dtypes, importlib-metadata, h5py, google-pasta, astunparse, requests-oauthlib, markdown, jax, google-auth, google-auth-oauthlib, tensorboard, tensorflow
  Attempting uninstall: numpy
    Found existing installation: numpy 1.25.2
    Uninstalling numpy-1.25.2:
      Successfully uninstalled numpy-1.25.2
Successfully installed MarkupSafe-2.1.3 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 flatbuffers-23.5.26 gast-0.4.0 google-auth-2.22.0 google-auth-oauthlib-1.0.0 google-pasta-0.2.0 grpcio-1.57.0 h5py-3.9.0 idna-3.4 importlib-metadata-6.8.0 jax-0.4.14 keras-2.12.0 libclang-16.0.6 markdown-3.4.4 ml-dtypes-0.2.0 numpy-1.23.5 oauthlib-3.2.2 opt-einsum-3.3.0 packaging-23.1 protobuf-4.24.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tensorboard-2.12.3 tensorboard-data-server-0.7.1 tensorflow-2.12.0 tensorflow-estimator-2.12.0 tensorflow-io-gcs-filesystem-0.33.0 termcolor-2.3.0 typing-extensions-4.7.1 urllib3-1.26.16 werkzeug-2.3.7 wrapt-1.14.1 zipp-3.16.2
$ mamba list | grep numpy
numpy                     1.23.5                   pypi_0    pypi
$ mamba list | grep scipy
scipy                     1.9.3            py39hddc5342_2    conda-forge
$ python -c "import scipy.linalg" # OK
$ python -c "import tensorflow; import scipy.ndimage"
2023-08-22 16:18:14.386624: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-22 16:18:21.059446: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/ndimage/__init__.py", line 153, in <module>
    from ._interpolation import *  # noqa: F401 F403
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/ndimage/_interpolation.py", line 37, in <module>
    from scipy import special
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/__init__.py", line 211, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/special/__init__.py", line 657, in <module>
    from . import _orthogonal
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/special/_orthogonal.py", line 81, in <module>
    from scipy import linalg
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/__init__.py", line 211, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/linalg/__init__.py", line 209, in <module>
    from ._matfuncs import *
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/linalg/_matfuncs.py", line 19, in <module>
    from ._matfuncs_sqrtm import sqrtm
  File "/path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/linalg/_matfuncs_sqrtm.py", line 24, in <module>
    from ._matfuncs_sqrtm_triu import within_block_loop
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /path/to/home/.conda/envs/tmp/lib/python3.9/site-packages/scipy/linalg/_matfuncs_sqrtm_triu.cpython-39-x86_64-linux-gnu.so)

It seems that pip installing tensorflow causes pip to downgrade the numpy installation (I don't know if this is related). After importing tensorflow, scipy.linalg appears to lookup the system libstdc++.so.6 rather than that installed by the libstdcxx-ng conda package.

$ uname -rs
Linux 5.10.179-166.674.amzn2.x86_64
rsanchezgarc commented 1 year ago

Hi, Can you try the following yaml instead? I just moved everything that uses numpy to the pip section

channels:
  - conda-forge
  - defaults
  - nvidia
dependencies:
  - cudatoolkit=11.8
  - h5py=3.9
  - hdf5=1.14
  - joblib=1.3
  - pip=23.1
  - python=3.9
  - requests=2.31
  - ruamel.yaml=0.17
  - tqdm=4.65
  - yaml=0.2
  - conda-build=3.25
  - pip:
      - absl-py==1.4.0
      - astunparse==1.6.3
      - flatbuffers==23.5.26
      - gast==0.4.0
      - google-auth==2.21.0
      - google-auth-oauthlib==1.0.0
      - google-pasta==0.2.0
      - grpcio==1.56.0
      - jax==0.4.13
      - keras==2.12.0
      - libclang==16.0.0
      - markdown==3.4.3
      - ml-dtypes==0.2.0
      - mrcfile==1.4
      - numpy==1.23
      - nvidia-cudnn-cu11==8.6.0.163
      - oauthlib==3.2.2
      - opt-einsum==3.3.0
      - protobuf==4.23.3
      - pyasn1==0.5.0
      - pyasn1-modules==0.3.0
      - requests-oauthlib==1.3.1
      - rsa==4.9
      - scikit-image==0.20
      - scipy==1.9
      - tensorboard==2.12.3
      - tensorboard-data-server==0.7.1
      - tensorflow==2.12.0
      - tensorflow-estimator==2.12.0
      - tensorflow-io-gcs-filesystem==0.32.0
      - termcolor==2.3.0
      - typing-extensions==4.7.0
env_vars:
  CUDNN_PATH: $CONDA_PREFIX/lib/python3.9/site-packages/nvidia/cudnn/
  LD_LIBRARY_PATH: $CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH
rjgildea commented 1 year ago

Thanks, I can confirm that the deepEMhancer now installs correctly on our systems using the above environment yaml file, and deepemhancer --download proceeds as expected 👍

rsanchezgarc commented 1 year ago

Great! I just updated the yml file in the repo. I am closing this

rsanchezgarc commented 1 year ago

I am opening the issue again because I noticed that this new installation yml does not properly configure the LD_LIBRARY_PATH to use cuda. I am working on this, but you can use the following (after activating the environment) to circumvent the issue

CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH
rsanchezgarc commented 1 year ago

In order to make the Library paths permanently available you can set them with conda env config vars set LD_LIBRARY_PATH=${CONDA_PREFIX}/lib/python3.9/site-packages/nvidia/cudnn/lib:${CONDA_PREFIX}/lib/:$LD_LIBRARY_PATH

I am afraid that this is the supported way of installing TensorFlow nowadays. I am updating the readme and closing this

rjgildea commented 1 year ago

With the original suggested changes to the yaml I was seeing ~162s/it in the in neural net inference step (presumably the inference was running on CPU?). After updating the repo to include your latest yaml changes and following the additional steps in the README to set the CUDA LD_LIBRARY_PATH I now get ~1.25s/it - much better! Thanks for your help!