Open carrisonj opened 2 days ago
Have you tried following the tips given at https://numpy.org/devdocs/user/troubleshooting-importerror.html?
I think this is an issue with the host you are running the docker container on, rather than the docker image itself or numpy.
I think https://stackoverflow.com/questions/52026652/openblas-blas-thread-init-pthread-create-resource-temporarily-unavailable is relevant.
either
should help. I don't know where you are running this, but if on an HPC facility, you might need assistance from the sysadmins for ulimit. Running on head (login) nodes is typically much more restricted than running on a compute node.
thank you, jsspencer,
i've add lines below to run_alphafold.py,
import os
os.environ['OPENBLAS_NUM_THREADS'] = '1'
before
import numpy as np
but got same error, how to fix the error ? more error msg as follow:
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
Traceback (most recent call last):
File "/alphafold3_venv/lib/python3.11/site-packages/numpy/_core/__init__.py", line 23, in <module>
from . import multiarray
File "/alphafold3_venv/lib/python3.11/site-packages/numpy/_core/multiarray.py", line 10, in <module>
from . import overrides
File "/alphafold3_venv/lib/python3.11/site-packages/numpy/_core/overrides.py", line 8, in <module>
from numpy._core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/alphafold3_venv/lib/python3.11/site-packages/numpy/__init__.py", line 114, in <module>
from numpy.__config__ import show as show_config
File "/alphafold3_venv/lib/python3.11/site-packages/numpy/__config__.py", line 4, in <module>
from numpy._core._multiarray_umath import (
File "/alphafold3_venv/lib/python3.11/site-packages/numpy/_core/__init__.py", line 49, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.11 from "/alphafold3_venv/bin/python3"
* The NumPy version is: "2.1.3"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: PyCapsule_Import could not import module "datetime"
Could you try running the docker run
command with the --privileged
flag?
add line
ENV OPENBLAS_NUM_THREADS=1
before
CMD ["python3", "run_alphafold.py"]
in Dockerfile, than rebuild and run image, got new error as follwo:
[root@node22 ~]# docker run -it --volume $HOME/af_input:/root/af_input --volume $HOME/af_output:/root/af_output --volume $MODEL_PARAMETERS_DIR:/root/models --volume $DB_DIR:/root/public_databases --gpus all alphafold3 python3 run_alphafold.py --json_path=/root/af_input/fold_input.json --model_dir=/root/models --output_dir=/root/af_output
I1121 09:37:49.169083 140327870522496 folding_input.py:1044] Detected /root/af_input/fold_input.json is an AlphaFold 3 JSON since the top-level is not a list.
Running AlphaFold 3. Please note that standard AlphaFold 3 model parameters are
only available under terms of use provided at
https://github.com/google-deepmind/alphafold3/blob/main/WEIGHTS_TERMS_OF_USE.md.
If you do not agree to these terms and are using AlphaFold 3 derived model
parameters, cancel execution of AlphaFold 3 inference with CTRL-C, and do not
use the model parameters.
2024-11-21 09:37:49.186558: F external/tsl/tsl/platform/default/env.cc:74] Check failed: ret == 0 (1 vs. 0)Thread tf_XLAEigen creation via pthread_create() failed.
Fatal Python error: Aborted
Current thread 0x00007fa0a0dfe480 (most recent call first):
File "/alphafold3_venv/lib/python3.11/site-packages/jaxlib/xla_client.py", line 76 in make_cpu_client
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 261 in make_cpu_client
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 971 in _init_backend
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 885 in backends
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 996 in _get_backend_uncached
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 1017 in get_backend
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 1141 in local_devices
File "/app/alphafold/run_alphafold.py", line 639 in main
File "/alphafold3_venv/lib/python3.11/site-packages/absl/app.py", line 254 in _run_main
File "/alphafold3_venv/lib/python3.11/site-packages/absl/app.py", line 308 in run
File "/app/alphafold/run_alphafold.py", line 674 in <module>
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, rdkit.rdBase, rdkit.DataStructs.cDataStructs, rdkit.Chem.rdchem, rdkit.Geometry.rdGeometry, rdkit.Chem.rdinchi, rdkit.Chem.rdCIPLabeler, rdkit.Chem.rdmolfiles, rdkit.Chem.rdmolops, rdkit.Chem.rdMolInterchange, rdkit.Chem.rdCoordGen, jaxlib.cpu_feature_guard, zstandard.backend_c, rdkit.ForceField.rdForceField, rdkit.Chem.rdChemicalFeatures, rdkit.Chem.rdMolChemicalFeatures, rdkit.Chem.rdDistGeom, rdkit.Chem.rdChemReactions, rdkit.Chem.rdDepictor, rdkit.Chem.rdFingerprintGenerator, rdkit.Chem.rdForceFieldHelpers, rdkit.Chem.rdMolAlign, rdkit.Chem.rdMolDescriptors, rdkit.Chem.rdMolEnumerator, rdkit.Chem.rdMolTransforms, rdkit.Chem.rdPartialCharges, rdkit.Chem.rdqueries, rdkit.Chem.rdReducedGraphs, rdkit.Chem.rdShapeHelpers, rdkit.Chem.rdSLNParse, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation (total: 79)
Fatal Python error: Segmentation fault
Current thread 0x00007fa0a0dfe480 (most recent call first):
File "/alphafold3_venv/lib/python3.11/site-packages/jaxlib/xla_client.py", line 76 in make_cpu_client
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 261 in make_cpu_client
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 971 in _init_backend
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 885 in backends
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 996 in _get_backend_uncached
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 1017 in get_backend
File "/alphafold3_venv/lib/python3.11/site-packages/jax/_src/xla_bridge.py", line 1141 in local_devices
File "/app/alphafold/run_alphafold.py", line 639 in main
File "/alphafold3_venv/lib/python3.11/site-packages/absl/app.py", line 254 in _run_main
File "/alphafold3_venv/lib/python3.11/site-packages/absl/app.py", line 308 in run
File "/app/alphafold/run_alphafold.py", line 674 in <module>
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, rdkit.rdBase, rdkit.DataStructs.cDataStructs, rdkit.Chem.rdchem, rdkit.Geometry.rdGeometry, rdkit.Chem.rdinchi, rdkit.Chem.rdCIPLabeler, rdkit.Chem.rdmolfiles, rdkit.Chem.rdmolops, rdkit.Chem.rdMolInterchange, rdkit.Chem.rdCoordGen, jaxlib.cpu_feature_guard, zstandard.backend_c, rdkit.ForceField.rdForceField, rdkit.Chem.rdChemicalFeatures, rdkit.Chem.rdMolChemicalFeatures, rdkit.Chem.rdDistGeom, rdkit.Chem.rdChemReactions, rdkit.Chem.rdDepictor, rdkit.Chem.rdFingerprintGenerator, rdkit.Chem.rdForceFieldHelpers, rdkit.Chem.rdMolAlign, rdkit.Chem.rdMolDescriptors, rdkit.Chem.rdMolEnumerator, rdkit.Chem.rdMolTransforms, rdkit.Chem.rdPartialCharges, rdkit.Chem.rdqueries, rdkit.Chem.rdReducedGraphs, rdkit.Chem.rdShapeHelpers, rdkit.Chem.rdSLNParse, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation (total: 79)
Looks similar to https://github.com/tensorflow/tensorflow/issues/41532.
Some things to help investigating this:
run docker build -t alphafold3 -f docker/Dockerfile . successfully, alphafold3 docker image created as follow:
but docker run failed with error as follow:
Plz help us solve it, thanks a lot.