google-deepmind / alphafold3

AlphaFold 3 inference pipeline.
Other
4.85k stars 524 forks source link

ERROR: fatal : Unsupported .version 8.4; current version is '8.3' #68

Open XIANZHE-LI opened 4 hours ago

XIANZHE-LI commented 4 hours ago

The following error occurred, do not know how to solve it

Running model inference for seed 123467... E1118 09:27:10.864007 36855 pjrt_stream_executor_client.cc:3084] Execution of replica 0 failed: INTERNAL: CustomCall failed: ptxas exited with non-zero error code 65280, output: ptxas /tmp/tempfile-911e745ba31e-6fcbb92-36855-62725d0d028be, line 5; fatal : Unsupported .version 8.4; current version is '8.3' ptxas fatal : Ptx assembly aborted due to errors

Traceback (most recent call last): File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 678, in app.run(main) File "/root/miniconda/envs/af3/lib/python3.11/site-packages/absl/app.py", line 308, in run _run_main(main, args) File "/root/miniconda/envs/af3/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) ^^^^^^^^^^ File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 663, in main process_fold_input( File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 542, in process_fold_input all_inference_results = predict_structure( ^^^^^^^^^^^^^^^^^^ File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 375, in predict_structure result = model_runner.run_inference(example, rng_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 313, in run_inference result = self._model(rng_key, featurised_example) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ jaxlib.xla_extension.XlaRuntimeError: INTERNAL: CustomCall failed: ptxas exited with non-zero error code 65280, output: ptxas /tmp/tempfile-911e745ba31e-6fcbb92-36855-62725d0d028be, line 5; fatal : Unsupported .version 8.4; current version is '8.3' ptxas fatal : Ptx assembly aborted due to errors


For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

XIANZHE-LI commented 2 hours ago

`# packages in environment at /root/miniconda/envs/af3: #

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge _openmp_mutex 4.5 2_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge absl-py 2.1.0 pypi_0 pypi alphafold3 3.0.0 pypi_0 pypi bzip2 1.0.8 h5eee18b_6 defaults ca-certificates 2024.9.24 h06a4308_0 defaults chex 0.1.87 pypi_0 pypi cudatoolkit 9.1 h85f986d_0 numba dm-haiku 0.0.13 pypi_0 pypi dm-tree 0.1.8 pypi_0 pypi filelock 3.16.1 pypi_0 pypi hmmer 3.4 hdbdd923_2 bioconda jax 0.4.34 pypi_0 pypi jax-cuda12-pjrt 0.4.34 pypi_0 pypi jax-cuda12-plugin 0.4.34 pypi_0 pypi jax-triton 0.2.0 pypi_0 pypi jaxlib 0.4.34 pypi_0 pypi jaxtyping 0.2.34 pypi_0 pypi jmp 0.0.4 pypi_0 pypi ld_impl_linux-64 2.40 h12ee557_0 defaults libffi 3.4.4 h6a678d5_1 defaults libgcc 14.2.0 h77fa898_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libgcc-ng 14.2.0 h69a702a_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libgomp 14.2.0 h77fa898_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libstdcxx 14.2.0 hc0a3c3a_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libstdcxx-ng 14.2.0 h4852527_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libuuid 1.41.5 h5eee18b_0 defaults ml-dtypes 0.5.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 defaults numpy 2.1.3 pypi_0 pypi nvidia-cublas-cu12 12.6.3.3 pypi_0 pypi nvidia-cuda-cupti-cu12 12.6.80 pypi_0 pypi nvidia-cuda-nvcc-cu12 12.6.77 pypi_0 pypi nvidia-cuda-runtime-cu12 12.6.77 pypi_0 pypi nvidia-cudnn-cu12 9.5.1.17 pypi_0 pypi nvidia-cufft-cu12 11.3.0.4 pypi_0 pypi nvidia-cusolver-cu12 11.7.1.2 pypi_0 pypi nvidia-cusparse-cu12 12.5.4.2 pypi_0 pypi nvidia-nccl-cu12 2.23.4 pypi_0 pypi nvidia-nvjitlink-cu12 12.6.77 pypi_0 pypi openssl 3.4.0 hb9d3cd8_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge opt-einsum 3.4.0 pypi_0 pypi pillow 11.0.0 pypi_0 pypi pip 24.2 py311h06a4308_0 defaults python 3.11.10 he870216_0 defaults rdkit 2024.3.5 pypi_0 pypi readline 8.2 h5eee18b_0 defaults scipy 1.14.1 pypi_0 pypi setuptools 75.1.0 py311h06a4308_0 defaults sqlite 3.45.3 h5eee18b_0 defaults tabulate 0.9.0 pypi_0 pypi tk 8.6.14 h39e8969_0 defaults toolz 1.0.0 pypi_0 pypi tqdm 4.67.0 pypi_0 pypi triton 3.1.0 pypi_0 pypi typeguard 2.13.3 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tzdata 2024b h04d1e81_0 defaults wheel 0.44.0 py311h06a4308_0 defaults xz 5.4.6 h5eee18b_1 defaults zlib 1.2.13 h5eee18b_1 defaults zstandard 0.23.0 pypi_0 pypi `

XIANZHE-LI commented 2 hours ago

nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Wed_Nov_22_10:17:15_PST_2023 Cuda compilation tools, release 12.3, V12.3.107 Build cuda_12.3.r12.3/compiler.33567101_0

XIANZHE-LI commented 2 hours ago

^C (af3) root@911e745ba31e:~/lanyun-tmp/AF3/alphafold3# nvidia-smi Mon Nov 18 11:53:01 2024
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4090 On | 00000000:3D:00.0 Off | Off | | 30% 28C P8 14W / 450W | 1MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+