search

Psy-Fer / deeplexicon

Signal based nanopore RNA demultiplexing with convolutional neural networks
https://psy-fer.github.io/deeplexicon/
MIT License
34 stars 8 forks source link

ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found #14

Closed Qiongyi closed 1 year ago

Qiongyi commented 3 years ago

Hi James,

I opened here a new issue and closed the previous one because we have a different error message using Python 3.7.

I am now using Python 3.7.5. All dependencies look OK, but I came across the `GLIBC_2.23' version issue. I think it is related to "tensorflow" and there is a github thread discussed this issue: https://github.com/tensorflow/tensorflow/issues/28660 They solved this issue by downgrading the Python to v3.6.8. This is the reason why I rolled back to Python 3.6.8 in the previous discussion: https://github.com/Psy-Fer/deeplexicon/issues/13

My command:

python deeplexicon.py -p 20201219_0616_MC-110494_0_FAK95916_1e4f2d65/fast5_pass -f multi -m models/resnet20-final.h5 > output.tsv

Error messages

Traceback (most recent call last): File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.5/lib/python3.7/imp.py", line 242, in load_module return load_dynamic(name, filename, file) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.5/lib/python3.7/imp.py", line 342, in load_dynamic return _load(spec) ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/illumina/tools/deeplexicon/deeplexicon.py", line 21, in import tensorflow as tf File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/init.py", line 24, in from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.5/lib/python3.7/imp.py", line 242, in load_module return load_dynamic(name, filename, file) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.5/lib/python3.7/imp.py", line 342, in load_dynamic return _load(spec) ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace above this error message when asking for help. (deeplexicon) [uqqzhao@inode2 deeplexicon]$ string /lib64/libm.so.6 |grep GLIBC bash: string: command not found... (deeplexicon) [uqqzhao@inode2 deeplexicon]$ strings /lib64/libm.so.6 |grep GLIBC GLIBC_2.2.5 GLIBC_2.4 GLIBC_2.15 GLIBC_PRIVATE GLIBC_2.15 strtold_nan@@GLIBC_PRIVATE errno_location@@GLIBC_2.2.5 errno@@GLIBC_PRIVATE qsort@@GLIBC_2.2.5 strtod_nan@@GLIBC_PRIVATE fputs@@GLIBC_2.2.5 memset@@GLIBC_2.2.5 feupdateenv@@GLIBC_2.2.5 GLIBC_2.4 _rtld_global_ro@@GLIBC_PRIVATE strtof_nan@@GLIBC_PRIVATE memmove@@GLIBC_2.2.5 fwrite@@GLIBC_2.2.5 __cxa_finalize@@GLIBC_2.2.5 stderr@@GLIBC_2.2.5

Dependencies

(deeplexicon) [uqqzhao@inode2 deeplexicon]$ pip install Keras==2.2.4 Pandas PyTs==0.8.0 Scikit-learn numba==0.45.0 TensorFlow==1.13.1 Requirement already satisfied: Keras==2.2.4 in ./lib/python3.7/site-packages (2.2.4) Requirement already satisfied: Pandas in ./lib/python3.7/site-packages (1.2.0) Requirement already satisfied: PyTs==0.8.0 in ./lib/python3.7/site-packages (0.8.0) Requirement already satisfied: Scikit-learn in ./lib/python3.7/site-packages (0.24.0) Requirement already satisfied: numba==0.45.0 in ./lib/python3.7/site-packages (0.45.0) Requirement already satisfied: TensorFlow==1.13.1 in ./lib/python3.7/site-packages (1.13.1) Requirement already satisfied: keras-applications>=1.0.6 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.0.8) Requirement already satisfied: keras-preprocessing>=1.0.5 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.1.2) Requirement already satisfied: numpy>=1.9.1 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.19.5) Requirement already satisfied: h5py in ./lib/python3.7/site-packages (from Keras==2.2.4) (3.1.0) Requirement already satisfied: pyyaml in ./lib/python3.7/site-packages (from Keras==2.2.4) (5.3.1) Requirement already satisfied: scipy>=0.14 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.6.0) Requirement already satisfied: six>=1.9.0 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.15.0) Requirement already satisfied: llvmlite>=0.29.0dev0 in ./lib/python3.7/site-packages (from numba==0.45.0) (0.32.1) Requirement already satisfied: absl-py>=0.1.6 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.11.0) Requirement already satisfied: grpcio>=1.8.6 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.34.0) Requirement already satisfied: termcolor>=1.1.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.1.0) Requirement already satisfied: tensorboard<1.14.0,>=1.13.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.13.1) Requirement already satisfied: tensorflow-estimator<1.14.0rc0,>=1.13.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.13.0) Requirement already satisfied: gast>=0.2.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.4.0) Requirement already satisfied: astor>=0.6.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.8.1) Requirement already satisfied: wheel>=0.26 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.36.2) Requirement already satisfied: protobuf>=3.6.1 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (3.14.0) Requirement already satisfied: markdown>=2.6.8 in ./lib/python3.7/site-packages (from tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.3.3) Requirement already satisfied: werkzeug>=0.11.15 in ./lib/python3.7/site-packages (from tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (1.0.1) Requirement already satisfied: importlib-metadata in ./lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.4.0) Requirement already satisfied: mock>=2.0.0 in ./lib/python3.7/site-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0->TensorFlow==1.13.1) (4.0.3) Requirement already satisfied: python-dateutil>=2.7.3 in ./lib/python3.7/site-packages (from Pandas) (2.8.1) Requirement already satisfied: pytz>=2017.3 in ./lib/python3.7/site-packages (from Pandas) (2020.5) Requirement already satisfied: threadpoolctl>=2.0.0 in ./lib/python3.7/site-packages (from Scikit-learn) (2.1.0) Requirement already satisfied: joblib>=0.11 in ./lib/python3.7/site-packages (from Scikit-learn) (1.0.0) Requirement already satisfied: cached-property in ./lib/python3.7/site-packages (from h5py->Keras==2.2.4) (1.5.2) Requirement already satisfied: typing-extensions>=3.6.4 in ./lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.7.4.3) Requirement already satisfied: zipp>=0.5 in ./lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.4.0)

Thanks,

Qiongyi

Qiongyi commented 3 years ago

Update:

After using Python 3.7.9, I still got the same GLIBC error.

Command & error messages

(deeplexicon) [uqqzhao@inode2 deeplexicon]$ python deeplexicon.py -p 20201219_0616_MC-110494_0_FAK95916_1e4f2d65/fast5_pass -f multi -m models/resnet20-final.h5 > output.tsv Traceback (most recent call last): File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.9/lib/python3.7/imp.py", line 242, in load_module return load_dynamic(name, filename, file) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.9/lib/python3.7/imp.py", line 342, in load_dynamic return _load(spec) ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "deeplexicon.py", line 21, in import tensorflow as tf File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/init.py", line 24, in from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.9/lib/python3.7/imp.py", line 242, in load_module return load_dynamic(name, filename, file) File "/clusterdata/uqqzhao/.pyenv/versions/3.7.9/lib/python3.7/imp.py", line 342, in load_dynamic return _load(spec) ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /illumina/tools/deeplexicon/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

Dependencies

(deeplexicon) [uqqzhao@inode2 deeplexicon]$ pip install Keras==2.2.4 Pandas PyTs==0.8.0 Scikit-learn numba==0.45.0 TensorFlow==1.13.1 Requirement already satisfied: Keras==2.2.4 in ./lib/python3.7/site-packages (2.2.4) Requirement already satisfied: Pandas in ./lib/python3.7/site-packages (1.2.0) Requirement already satisfied: PyTs==0.8.0 in ./lib/python3.7/site-packages (0.8.0) Requirement already satisfied: Scikit-learn in ./lib/python3.7/site-packages (0.24.0) Requirement already satisfied: numba==0.45.0 in ./lib/python3.7/site-packages (0.45.0) Requirement already satisfied: TensorFlow==1.13.1 in ./lib/python3.7/site-packages (1.13.1) Requirement already satisfied: pyyaml in ./lib/python3.7/site-packages (from Keras==2.2.4) (5.3.1) Requirement already satisfied: scipy>=0.14 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.6.0) Requirement already satisfied: six>=1.9.0 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.15.0) Requirement already satisfied: h5py in ./lib/python3.7/site-packages (from Keras==2.2.4) (3.1.0) Requirement already satisfied: keras-applications>=1.0.6 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.0.8) Requirement already satisfied: numpy>=1.9.1 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.19.5) Requirement already satisfied: keras-preprocessing>=1.0.5 in ./lib/python3.7/site-packages (from Keras==2.2.4) (1.1.2) Requirement already satisfied: llvmlite>=0.29.0dev0 in ./lib/python3.7/site-packages (from numba==0.45.0) (0.32.1) Requirement already satisfied: astor>=0.6.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.8.1) Requirement already satisfied: tensorboard<1.14.0,>=1.13.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.13.1) Requirement already satisfied: protobuf>=3.6.1 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (3.14.0) Requirement already satisfied: tensorflow-estimator<1.14.0rc0,>=1.13.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.13.0) Requirement already satisfied: absl-py>=0.1.6 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.11.0) Requirement already satisfied: termcolor>=1.1.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.1.0) Requirement already satisfied: grpcio>=1.8.6 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (1.34.0) Requirement already satisfied: wheel>=0.26 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.36.2) Requirement already satisfied: gast>=0.2.0 in ./lib/python3.7/site-packages (from TensorFlow==1.13.1) (0.4.0) Requirement already satisfied: werkzeug>=0.11.15 in ./lib/python3.7/site-packages (from tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (1.0.1) Requirement already satisfied: markdown>=2.6.8 in ./lib/python3.7/site-packages (from tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.3.3) Requirement already satisfied: importlib-metadata in ./lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.4.0) Requirement already satisfied: mock>=2.0.0 in ./lib/python3.7/site-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0->TensorFlow==1.13.1) (4.0.3) Requirement already satisfied: python-dateutil>=2.7.3 in ./lib/python3.7/site-packages (from Pandas) (2.8.1) Requirement already satisfied: pytz>=2017.3 in ./lib/python3.7/site-packages (from Pandas) (2020.5) Requirement already satisfied: joblib>=0.11 in ./lib/python3.7/site-packages (from Scikit-learn) (1.0.0) Requirement already satisfied: threadpoolctl>=2.0.0 in ./lib/python3.7/site-packages (from Scikit-learn) (2.1.0) Requirement already satisfied: cached-property in ./lib/python3.7/site-packages (from h5py->Keras==2.2.4) (1.5.2) Requirement already satisfied: typing-extensions>=3.6.4 in ./lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.7.4.3) Requirement already satisfied: zipp>=0.5 in ./lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->TensorFlow==1.13.1) (3.4.0) (deeplexicon) [uqqzhao@inode2 deeplexicon]$ python --version Python 3.7.9 (deeplexicon) [uqqzhao@inode2 deeplexicon]$ lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.7.1908 (Core) Release: 7.7.1908 Codename: Core

Psy-Fer commented 3 years ago

This can probably be solved with the docker container @noncodo created. Care to share Martin?

enovoa commented 3 years ago

HI @Psy-Fer i think you might be referring to the Dockerfile that we created? (although perhaps @noncodo also made one?) In any case, ours is here: https://github.com/novoalab/DeePlexiCon/blob/master/Dockerfile

Psy-Fer commented 3 years ago

Ahh yes that's the one. Cheers

lpryszcz commented 3 years ago

the docker image

docker run -u $UID:$GID -v path_to_fast5:/data biocorecrg/deeplexicon:0.1 deeplexicon/deeplexicon.py -p /data -f multi -m deeplexicon/models/resnet20-final.h5 > demux.tsv
DepledgeLab commented 2 years ago

Hello, has there been any advance in solving this issue? We are not able to use docker on our local HPCs and I have been unable to get deeplexicon working in python environment (same issue as above with GLIBC_2.23')?

Would appreciate any advice as I have a lot of data that is sitting and waiting for demultiplexing...

enovoa commented 2 years ago

Hi @DepledgeLab - sorry, we have for the moment we are mainly using DeePlexiCon through docker, either as a standalone image (see above) or via MasterOfPores NextFlow workflow (which also uses Docker or Singularity, see: https://github.com/biocorecrg/master_of_pores). If you can't use docker in your local HPCs, please note that you can also launch MasterOfPores (which includes deeplexicon) in Amazon Cloud (see Section "Running MasterOfPores on the Cloud (AWS Batch and AWS EC2" in this paper for more details: https://www.frontiersin.org/articles/10.3389/fgene.2020.00211/full). Hope that helps!

DepledgeLab commented 2 years ago

Thanks for the quick response. I managed to get it working through singularity although I would note the standalone image noted above does not install properly. I was able to get it working using this image - https://hub.docker.com/r/lpryszcz/deeplexicon although it required reinstalling tensorflow after the fact.

I can't seem to get the GPU side working at present (GPU not detected, please ensure Drivers/CUDA/cuDNN/tf-gpu are set up properly) so would appreciate any advice on how to set up these drivers within Singularity.

enovoa commented 2 years ago

Hi @DepledgeLab, glad to know that the solution of using Singularity worked for you, and thanks for pointing out an alternative Docker image. About having issues using GPU within Singularity, in our case there was also some troubleshooting involved, but the exact issues that one finds will vary depending on your own GPU, CUDA and singularity versions.... I would recommend taking a look at this doc: https://docs.sylabs.io/guides/3.5/user-guide/gpu.html , and if you still have issues to get it to work, you may want to join the Singularity Slack group (https://hpcng.slack.com), based on their website they mention this is the way to get quick feedback. Hope that helps!

DepledgeLab commented 2 years ago

Thanks - this is very helpful.

I'm also experimenting with multi-threading on the CPU which could work as an effective workaround. Is there a hard-coded limit on the number of threads that can be used by deep lexicon?

Psy-Fer commented 2 years ago

It depends on which version/script of deeplexicon you are using. If it is one with multiprocessing implemented, then no, there isn't a hard limit.