Open XIANZHE-LI opened 4 hours ago
`# packages in environment at /root/miniconda/envs/af3: #
_libgcc_mutex 0.1 conda_forge https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge _openmp_mutex 4.5 2_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge absl-py 2.1.0 pypi_0 pypi alphafold3 3.0.0 pypi_0 pypi bzip2 1.0.8 h5eee18b_6 defaults ca-certificates 2024.9.24 h06a4308_0 defaults chex 0.1.87 pypi_0 pypi cudatoolkit 9.1 h85f986d_0 numba dm-haiku 0.0.13 pypi_0 pypi dm-tree 0.1.8 pypi_0 pypi filelock 3.16.1 pypi_0 pypi hmmer 3.4 hdbdd923_2 bioconda jax 0.4.34 pypi_0 pypi jax-cuda12-pjrt 0.4.34 pypi_0 pypi jax-cuda12-plugin 0.4.34 pypi_0 pypi jax-triton 0.2.0 pypi_0 pypi jaxlib 0.4.34 pypi_0 pypi jaxtyping 0.2.34 pypi_0 pypi jmp 0.0.4 pypi_0 pypi ld_impl_linux-64 2.40 h12ee557_0 defaults libffi 3.4.4 h6a678d5_1 defaults libgcc 14.2.0 h77fa898_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libgcc-ng 14.2.0 h69a702a_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libgomp 14.2.0 h77fa898_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libstdcxx 14.2.0 hc0a3c3a_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libstdcxx-ng 14.2.0 h4852527_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge libuuid 1.41.5 h5eee18b_0 defaults ml-dtypes 0.5.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 defaults numpy 2.1.3 pypi_0 pypi nvidia-cublas-cu12 12.6.3.3 pypi_0 pypi nvidia-cuda-cupti-cu12 12.6.80 pypi_0 pypi nvidia-cuda-nvcc-cu12 12.6.77 pypi_0 pypi nvidia-cuda-runtime-cu12 12.6.77 pypi_0 pypi nvidia-cudnn-cu12 9.5.1.17 pypi_0 pypi nvidia-cufft-cu12 11.3.0.4 pypi_0 pypi nvidia-cusolver-cu12 11.7.1.2 pypi_0 pypi nvidia-cusparse-cu12 12.5.4.2 pypi_0 pypi nvidia-nccl-cu12 2.23.4 pypi_0 pypi nvidia-nvjitlink-cu12 12.6.77 pypi_0 pypi openssl 3.4.0 hb9d3cd8_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge opt-einsum 3.4.0 pypi_0 pypi pillow 11.0.0 pypi_0 pypi pip 24.2 py311h06a4308_0 defaults python 3.11.10 he870216_0 defaults rdkit 2024.3.5 pypi_0 pypi readline 8.2 h5eee18b_0 defaults scipy 1.14.1 pypi_0 pypi setuptools 75.1.0 py311h06a4308_0 defaults sqlite 3.45.3 h5eee18b_0 defaults tabulate 0.9.0 pypi_0 pypi tk 8.6.14 h39e8969_0 defaults toolz 1.0.0 pypi_0 pypi tqdm 4.67.0 pypi_0 pypi triton 3.1.0 pypi_0 pypi typeguard 2.13.3 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tzdata 2024b h04d1e81_0 defaults wheel 0.44.0 py311h06a4308_0 defaults xz 5.4.6 h5eee18b_1 defaults zlib 1.2.13 h5eee18b_1 defaults zstandard 0.23.0 pypi_0 pypi `
nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Wed_Nov_22_10:17:15_PST_2023 Cuda compilation tools, release 12.3, V12.3.107 Build cuda_12.3.r12.3/compiler.33567101_0
^C
(af3) root@911e745ba31e:~/lanyun-tmp/AF3/alphafold3# nvidia-smi
Mon Nov 18 11:53:01 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:3D:00.0 Off | Off |
| 30% 28C P8 14W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
The following error occurred, do not know how to solve it
Running model inference for seed 123467... E1118 09:27:10.864007 36855 pjrt_stream_executor_client.cc:3084] Execution of replica 0 failed: INTERNAL: CustomCall failed: ptxas exited with non-zero error code 65280, output: ptxas /tmp/tempfile-911e745ba31e-6fcbb92-36855-62725d0d028be, line 5; fatal : Unsupported .version 8.4; current version is '8.3' ptxas fatal : Ptx assembly aborted due to errors
Traceback (most recent call last): File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 678, in
app.run(main)
File "/root/miniconda/envs/af3/lib/python3.11/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/root/miniconda/envs/af3/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
^^^^^^^^^^
File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 663, in main
process_fold_input(
File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 542, in process_fold_input
all_inference_results = predict_structure(
^^^^^^^^^^^^^^^^^^
File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 375, in predict_structure
result = model_runner.run_inference(example, rng_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lanyun-tmp/AF3/alphafold3/run_alphafold.py", line 313, in run_inference
result = self._model(rng_key, featurised_example)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: CustomCall failed: ptxas exited with non-zero error code 65280, output: ptxas /tmp/tempfile-911e745ba31e-6fcbb92-36855-62725d0d028be, line 5; fatal : Unsupported .version 8.4; current version is '8.3'
ptxas fatal : Ptx assembly aborted due to errors
For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.