search

kalininalab / alphafold_non_docker

AlphaFold2 non-docker setup
346 stars 120 forks source link

AttributeError: module 'jax.errors' has no attribute 'UnexpectedTracerError' #57

Closed ponomarevsy closed 1 year ago

ponomarevsy commented 1 year ago

Hello,

I am using v2.3.1 of alphafold. I've installed it inside conda environment by combining your conda instructions with the https://github.com/deepmind/alphafold/archive/refs/tags/v2.3.1.tar.gz distro.

When I run a test example I am getting this error somewhere close to the end of the pipeline (which takes a few minutes to run):

...
I0120 11:25:59.991136 47767026135936 run_alphafold.py:191] Running model model_1_pred_0 on query
I0120 11:26:06.017625 47767026135936 model.py:165] Running predict with shape(feat) = {'aatype': (4, 68), 'residue_index': (4, 68), 'seq_length': (4,), 'template_aatype': (4, 4, 68), 'template_all_atom_masks': (4, 4, 68, 37), 'template_all_atom_positions': (4, 4, 68, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 68), 'msa_mask': (4, 508, 68), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_mask': (4, 4), 'template_pseudo_beta': (4, 4, 68, 3), 'template_pseudo_beta_mask': (4, 4, 68), 'atom14_atom_exists': (4, 68, 14), 'residx_atom14_to_atom37': (4, 68, 14), 'residx_atom37_to_atom14': (4, 68, 37), 'atom37_atom_exists': (4, 68, 37), 'extra_msa': (4, 5120, 68), 'extra_msa_mask': (4, 5120, 68), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 68), 'true_msa': (4, 508, 68), 'extra_has_deletion': (4, 5120, 68), 'extra_deletion_value': (4, 5120, 68), 'msa_feat': (4, 508, 68, 49), 'target_feat': (4, 68, 22)}
...
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/alphafold/2.3.1-Python-3.8.0/run_alphafold.py", line 432, in <module>
    app.run(main)
  File "/path/to/Anaconda3/2020.07/envs/alphafold-2.3.1/lib/python3.8/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/path/to/Anaconda3/2020.07/envs/alphafold-2.3.1/lib/python3.8/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/path/to/alphafold/2.3.1-Python-3.8.0/run_alphafold.py", line 408, in main
    predict_structure(
  File "/path/to/alphafold/2.3.1-Python-3.8.0/run_alphafold.py", line 199, in predict_structure
    prediction_result = model_runner.predict(processed_feature_dict,
  File "/path/to/alphafold/2.3.1-Python-3.8.0/alphafold/model/model.py", line 167, in predict
    result = self.apply(self.params, jax.random.PRNGKey(random_seed), feat)
  File "/path/to/Anaconda3/2020.07/envs/alphafold-2.3.1/lib/python3.8/site-packages/haiku/_src/transform.py", line 127, in apply_fn
    out, state = f.apply(params, {}, *args, **kwargs)
  File "/path/to/Anaconda3/2020.07/envs/alphafold-2.3.1/lib/python3.8/site-packages/haiku/_src/transform.py", line 384, in apply_fn
    except jax.errors.UnexpectedTracerError as e:
AttributeError: module 'jax.errors' has no attribute 'UnexpectedTracerError'

Does this have to do with incompatible jax/jaxlib versions? This is what I have right now:

(alphafold-2.3.1) [hostname]» conda list jax
# packages in environment at /path/to/Anaconda3/2020.07/envs/alphafold-2.3.1:
#
# Name                    Version                   Build  Channel
jax                       0.2.14                   pypi_0    pypi
jaxlib                    0.4.1                    pypi_0    pypi

Please let me know how to proceed. Thank you!

PasqM commented 1 year ago

Hi,

as you noticed this could be due to jax incompatibility, The right versions to run alphafold v2.3.1 should be jax==0.3.25 and jaxlib==0.3.25+cuda11.cudnn805.

ponomarevsy commented 1 year ago

Thank you, @PasqM! I am getting this error:

» pip install --upgrade jax==0.3.25 jaxlib==0.3.25+cuda11.cudnn805 -f https://sto
rage.googleapis.com/jax-releases/jax_releases.html
Looking in links: https://storage.googleapis.com/jax-releases/jax_releases.html
Collecting jax==0.3.25
  Downloading jax-0.3.25.tar.gz (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 16.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
ERROR: Could not find a version that satisfies the requirement jaxlib==0.3.25+cuda11.cudnn805 (from versions: 0.1.32, 0.1.40, 0.1.41, 0.1.42, 0.1.43, 0.1.44, 0.1.46, 0.1.50, 0.1.51, 0.1.52, 0.1.55, 0.1.56, 0.1.57, 0.1.58, 0.1.59, 0.1.60, 0.1.61, 0.1.62, 0.1.63, 0.1.64, 0.1.65, 0.1.66, 0.1.67, 0.1.68, 0.1.69, 0.1.70, 0.1.71, 0.1.72, 0.1.73, 0.1.74, 0.1.75, 0.1.76, 0.3.0, 0.3.2, 0.3.5, 0.3.7, 0.3.8, 0.3.10, 0.3.14, 0.3.15, 0.3.18, 0.3.20, 0.3.22, 0.3.24, 0.3.25, 0.4.0, 0.4.1)
ERROR: No matching distribution found for jaxlib==0.3.25+cuda11.cudnn805

If I remove the "+cuda" part it installs, but the pipeline won't see GPUs (I am running on a 8 x V100 GPU node):

...
I0123 10:14:56.065946 47820129538944 xla_bridge.py:353] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0123 10:14:56.066154 47820129538944 xla_bridge.py:353] Unable to initialize backend 'cuda': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
I0123 10:14:56.066246 47820129538944 xla_bridge.py:353] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
I0123 10:14:56.068324 47820129538944 xla_bridge.py:353] Unable to initialize backend 'tpu': INVALID_ARGUMENT: TpuPlatform is not available.
...
W0123 10:14:56.068625 47820129538944 xla_bridge.py:360] No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
...

Also, is there a newer version of 'run_alphafold.sh' by Sanjay Kumar Srikakulam I can download?

PasqM commented 1 year ago

Try pip install --upgrade --no-cache-dir jax==0.3.25 jaxlib==0.3.25+cuda11.cudnn805 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html, it should find the cuda version.

By the way, in the fork https://github.com/PasqM/alphafold_non_docker everything is updated to v2.3.1. It works in my set up and I submitted a pull request to this repo, try it if you like (and please report any issue).

ponomarevsy commented 1 year ago

And that 'pip' command worked indeed! Thank you. The "Unable to initialize backend 'cuda'" error is gone: 

I0123 10:35:50.465131 47549919085440 xla_bridge.py:353] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0123 10:35:50.901049 47549919085440 xla_bridge.py:353] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: Host Interpreter CUDA
I0123 10:35:50.901945 47549919085440 xla_bridge.py:353] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I0123 10:35:50.902230 47549919085440 xla_bridge.py:353] Unable to initialize backend 'plugin': xla_extension has no attributes named get_plugin_device_client. Compile TensorFlow with //tensorflow/compiler/xla/python:enable_plugin_device set to true (defaults to false) to enable this.

But now I see a different error with a crash:

jax._src.traceback_util.UnfilteredStackTrace: TypeError: __init__() got an unexpected keyword argument 'param_axis'

Does it have to do with the input file? Here is what I have now for jax:

» conda list jax                                           
# packages in environment at /path/to/Anaconda3/2020.07/envs/alphafold-2.3.1:
#
# Name                    Version                   Build  Channel
jax                       0.3.25                   pypi_0    pypi
jaxlib                    0.3.25+cuda11.cudnn805          pypi_0    pypi

» conda list cuda                                         
# packages in environment at /path/to/Anaconda3/2020.07/envs/alphafold-2.3.1:
#
# Name                    Version                   Build  Channel
cudatoolkit               11.2.2              hbe64b41_10    conda-forge

Please let me know if've seen this before...

PasqM commented 1 year ago

I've never seen this, but I suggest verifying that every package is in the version stated in the README of https://github.com/PasqM/alphafold_non_docker (should be the one needed by the latest version of AF).

ponomarevsy commented 1 year ago

Yes, thank you. Used the latest 'run_alphafold.sh' and still getting the same error:

  File "/path/to/alphafold/2.3.1-Python-3.8.0/alphafold/model/common_modules.py", line 150, in __init__
    super().__init__(
TypeError: __init__() got an unexpected keyword argument 'param_axis'
...

» grep -i -r param_axis .
./alphafold/model/common_modules.py:               param_axis=None):
./alphafold/model/common_modules.py:        param_axis=param_axis)
./alphafold/model/common_modules.py:    param_axis = self.param_axis[0] if self.param_axis else -1
./alphafold/model/common_modules.py:    param_shape = (x.shape[param_axis],)
./alphafold/model/common_modules.py:    param_broadcast_shape[param_axis] = x.shape[param_axis]
./alphafold/model/modules.py:      param_axis=axis,
ponomarevsy commented 1 year ago

Hi @PasqM,

Would you kindly tell me where to download multimer_query.fasta, monomer.fasta and homomer.fasta files that are used in the examples here: https://github.com/PasqM/alphafold_non_docker? I think I am getting closer to solving this mystery after starting from scratch. Thank you for your help, as always.

PasqM commented 1 year ago

Hi @ponomarevsy, these files are examples, you should replace them with the fasta file of your query sequence. Regarding your errors, I don't know what's the reason, I would try first of all to redownload alphafold 2.3.1 from https://github.com/deepmind/alphafold/archive/refs/tags/v2.3.1.tar.gz and then if it doesn't work to recreate the entire conda environment.

ponomarevsy commented 1 year ago

Started from scratch and I think it works now (some databases were missing in the previous step).

$ ./run_locally.bash
I0127 11:39:07.214764 47607713561088 templates.py:857] Using precomputed obsolete pdbs /path/to/alphafold database/pdb_mmcif/obsolete.dat.
I0127 11:39:07.451822 47607713561088 xla_bridge.py:353] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0127 11:39:07.714668 47607713561088 xla_bridge.py:353] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: Interpreter CUDA Host
I0127 11:39:07.715586 47607713561088 xla_bridge.py:353] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I0127 11:39:07.715865 47607713561088 xla_bridge.py:353] Unable to initialize backend 'plugin': xla_extension has no attributes named get_plugin_device_client. Compile TensorFlow with //tensorflow/compiler/xla/python:enable_plugin_device set to true (defaults to false) to enable this.
I0127 11:39:11.530140 47607713561088 run_alphafold.py:386] Have 5 models: ['model_1_pred_0', 'model_2_pred_0', 'model_3_pred_0', 'model_4_pred_0', 'model_5_pred_0']
I0127 11:39:11.530411 47607713561088 run_alphafold.py:403] Using random seed 1698875176206529255 for the data pipeline
I0127 11:39:11.530826 47607713561088 run_alphafold.py:161] Predicting query
I0127 11:39:11.533574 47607713561088 jackhmmer.py:133] Launching subprocess "path/to/Anaconda3/2022.05/envs/alphafold-2.3.1/bin/jackhmmer -o /dev/null -A /tmp/tmptuqx76m5/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /path/to/alphafold/2.3.1-Python-3.8.0/example/query.fasta /path/to/alphafold database/uniref90/uniref90.fasta"
I0127 11:39:11.588075 47607713561088 utils.py:36] Started Jackhmmer (uniref90.fasta) query
I0127 11:42:57.156229 47607713561088 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 225.568 seconds
I0127 11:42:57.160816 47607713561088 jackhmmer.py:133] Launching subprocess "/path/to/Anaconda3/2022.05/envs/alphafold-2.3.1/bin/jackhmmer -o /dev/null -A /tmp/tmphjfp3q0t/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /sysapps/cluster/software/alphafold/2.3.1-Python-3.8.0/example/query.fasta /path.to/alphafold database/mgnify/mgy_clusters_2022_05.fa"
I0127 11:42:57.222830 47607713561088 utils.py:36] Started Jackhmmer (mgy_clusters_2022_05.fa) query
I0127 11:51:12.557061 47607713561088 utils.py:40] Finished Jackhmmer (mgy_clusters_2022_05.fa) query in 495.334 seconds
I0127 11:51:12.562014 47607713561088 hhsearch.py:85] Launching subprocess "path/to/Anaconda3/2022.05/envs/alphafold-2.3.1/bin/hhsearch -i /tmp/tmpbjutb7tr/query.a3m -o /tmp/tmpbjutb7tr/output.hhr -maxseq 1000000 -d /path/to/alphafold database/pdb70/pdb70"
I0127 11:51:12.627545 47607713561088 utils.py:36] Started HHsearch query
I0127 11:51:35.905422 47607713561088 utils.py:40] Finished HHsearch query in 23.277 seconds
I0127 11:51:35.963973 47607713561088 hhblits.py:128] Launching subprocess "/path/to/Anaconda3/2022.05/envs/alphafold-2.3.1/bin/hhblits -i /sysapps/cluster/software/alphafold/2.3.1-Python-3.8.0/example/query.fasta -cpu 4 -oa3m /tmp/tmp47twq9d2/output.a3m -o /dev/null -n 3 -e 0.001 -maxseq 1000000 -realign_max 100000 -maxfilt 100000 -min_prefilter_hits 1000 -d /path/to/alphafold database/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt -d /path/to/alphafold database/uniref30/UniRef30_2021_03"
I0127 11:51:36.025832 47607713561088 utils.py:36] Started HHblits query
I0127 11:55:36.883327 47607713561088 utils.py:40] Finished HHblits query in 240.857 seconds
I0127 11:55:36.885067 47607713561088 templates.py:878] Searching for template for: GWSTELEKHREELKEFLKKEGITNVEIRIDNGRLEVRVEGGTERLKRFLEELRQKLEKKGYTVDIKIE
I0127 11:55:36.886693 47607713561088 templates.py:718] hit 6mrr_A did not pass prefilter: Template is an exact subsequence of query with large coverage. Length ratio: 1.0.
I0127 11:55:36.886936 47607713561088 templates.py:912] Skipped invalid hit 6MRR_A foldit1; De novo protein, Foldit; 1.18A {synthetic construct}, error: None, warning: None
I0127 11:55:37.177063 47607713561088 templates.py:267] Found an exact template match 6q64_A.
I0127 11:55:37.820456 47607713561088 templates.py:267] Found an exact template match 4s3k_A.
I0127 11:55:38.102424 47607713561088 templates.py:267] Found an exact template match 5jh8_A.
I0127 11:55:38.260325 47607713561088 templates.py:267] Found an exact template match 1jnd_A.
I0127 11:55:38.673694 47607713561088 templates.py:267] Found an exact template match 5y2a_B.
I0127 11:55:39.703439 47607713561088 templates.py:267] Found an exact template match 4wiw_B.
I0127 11:55:39.712453 47607713561088 templates.py:267] Found an exact template match 4wiw_D.
I0127 11:55:39.841529 47607713561088 templates.py:267] Found an exact template match 6jm7_A.
I0127 11:55:40.176994 47607713561088 templates.py:267] Found an exact template match 6jmb_A.
I0127 11:55:40.367680 47607713561088 templates.py:267] Found an exact template match 4q6t_A.
I0127 11:55:40.699988 47607713561088 templates.py:267] Found an exact template match 3oa5_B.
I0127 11:55:40.950988 47607713561088 templates.py:267] Found an exact template match 5y2c_A.
I0127 11:55:40.993796 47607713561088 templates.py:267] Found an exact template match 5cuk_A.
I0127 11:55:42.017782 47607713561088 templates.py:267] Found an exact template match 4a5q_E.
I0127 11:55:42.155383 47607713561088 templates.py:267] Found an exact template match 5y2b_A.
I0127 11:55:42.305338 47607713561088 templates.py:267] Found an exact template match 4lgx_A.
I0127 11:55:42.687880 47607713561088 templates.py:267] Found an exact template match 4w5u_A.
I0127 11:55:42.827735 47607713561088 templates.py:267] Found an exact template match 6jav_A.
I0127 11:55:43.341216 47607713561088 templates.py:267] Found an exact template match 3cz8_A.
I0127 11:55:43.349645 47607713561088 templates.py:267] Found an exact template match 3cz8_B.
I0127 11:55:43.358644 47607713561088 pipeline.py:234] Uniref90 MSA size: 2 sequences.
I0127 11:55:43.358720 47607713561088 pipeline.py:235] BFD MSA size: 1 sequences.
I0127 11:55:43.358756 47607713561088 pipeline.py:236] MGnify MSA size: 2 sequences.
I0127 11:55:43.358790 47607713561088 pipeline.py:237] Final (deduplicated) MSA size: 2 sequences.
I0127 11:55:43.358978 47607713561088 pipeline.py:239] Total number of templates (NB: this can include bad templates and is later filtered to top 4): 20.
I0127 11:55:43.363679 47607713561088 run_alphafold.py:191] Running model model_1_pred_0 on query
I0127 11:55:46.180077 47607713561088 model.py:165] Running predict with shape(feat) = {'aatype': (4, 68), 'residue_index': (4, 68), 'seq_length': (4,), 'template_aatype': (4, 4, 68), 'template_all_atom_masks': (4, 4, 68, 37), 'template_all_atom_positions': (4, 4, 68, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 68), 'msa_mask': (4, 508, 68), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_mask': (4, 4), 'template_pseudo_beta': (4, 4, 68, 3), 'template_pseudo_beta_mask': (4, 4, 68), 'atom14_atom_exists': (4, 68, 14), 'residx_atom14_to_atom37': (4, 68, 14), 'residx_atom37_to_atom14': (4, 68, 37), 'atom37_atom_exists': (4, 68, 37), 'extra_msa': (4, 5120, 68), 'extra_msa_mask': (4, 5120, 68), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 68), 'true_msa': (4, 508, 68), 'extra_has_deletion': (4, 5120, 68), 'extra_deletion_value': (4, 5120, 68), 'msa_feat': (4, 508, 68, 49), 'target_feat': (4, 68, 22)}
I0127 11:57:47.712098 47607713561088 model.py:175] Output shape was {'distogram': {'bin_edges': (63,), 'logits': (68, 68, 64)}, 'experimentally_resolved': {'logits': (68, 37)}, 'masked_msa': {'logits': (508, 68, 23)}, 'predicted_lddt': {'logits': (68, 50)}, 'structure_module': {'final_atom_mask': (68, 37), 'final_atom_positions': (68, 37, 3)}, 'plddt': (68,), 'ranking_confidence': ()}
I0127 11:57:47.712713 47607713561088 run_alphafold.py:203] Total JAX model model_1_pred_0 on query predict time (includes compilation time, see --benchmark): 121.5s
I0127 11:58:00.912315 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {<Residue 67 (GLU) of chain 0>: ['OXT']}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 11:58:01.006034 47607713561088 amber_minimize.py:408] Minimizing protein, attempt 1 of 100.
I0127 11:58:01.227611 47607713561088 amber_minimize.py:69] Restraining 574 / 1170 particles.
I0127 11:58:03.770199 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 11:58:05.229933 47607713561088 amber_minimize.py:500] Iteration completed: Einit 771.01 Efinal -2142.23 Time 2.00 s num residue violations 0 num residue exclusions 0
I0127 11:58:05.653247 47607713561088 run_alphafold.py:191] Running model model_2_pred_0 on query
I0127 11:58:07.032880 47607713561088 model.py:165] Running predict with shape(feat) = {'aatype': (4, 68), 'residue_index': (4, 68), 'seq_length': (4,), 'template_aatype': (4, 4, 68), 'template_all_atom_masks': (4, 4, 68, 37), 'template_all_atom_positions': (4, 4, 68, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 68), 'msa_mask': (4, 508, 68), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_mask': (4, 4), 'template_pseudo_beta': (4, 4, 68, 3), 'template_pseudo_beta_mask': (4, 4, 68), 'atom14_atom_exists': (4, 68, 14), 'residx_atom14_to_atom37': (4, 68, 14), 'residx_atom37_to_atom14': (4, 68, 37), 'atom37_atom_exists': (4, 68, 37), 'extra_msa': (4, 1024, 68), 'extra_msa_mask': (4, 1024, 68), 'extra_msa_row_mask': (4, 1024), 'bert_mask': (4, 508, 68), 'true_msa': (4, 508, 68), 'extra_has_deletion': (4, 1024, 68), 'extra_deletion_value': (4, 1024, 68), 'msa_feat': (4, 508, 68, 49), 'target_feat': (4, 68, 22)}
I0127 12:00:04.546868 47607713561088 model.py:175] Output shape was {'distogram': {'bin_edges': (63,), 'logits': (68, 68, 64)}, 'experimentally_resolved': {'logits': (68, 37)}, 'masked_msa': {'logits': (508, 68, 23)}, 'predicted_lddt': {'logits': (68, 50)}, 'structure_module': {'final_atom_mask': (68, 37), 'final_atom_positions': (68, 37, 3)}, 'plddt': (68,), 'ranking_confidence': ()}
I0127 12:00:04.547415 47607713561088 run_alphafold.py:203] Total JAX model model_2_pred_0 on query predict time (includes compilation time, see --benchmark): 117.5s
I0127 12:00:08.543631 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {<Residue 67 (GLU) of chain 0>: ['OXT']}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:00:08.651145 47607713561088 amber_minimize.py:408] Minimizing protein, attempt 1 of 100.
I0127 12:00:08.860064 47607713561088 amber_minimize.py:69] Restraining 574 / 1170 particles.
I0127 12:00:10.645244 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:00:10.823759 47607713561088 amber_minimize.py:500] Iteration completed: Einit 732.75 Efinal -2225.50 Time 1.03 s num residue violations 0 num residue exclusions 0
I0127 12:00:11.419507 47607713561088 run_alphafold.py:191] Running model model_3_pred_0 on query
I0127 12:00:12.611460 47607713561088 model.py:165] Running predict with shape(feat) = {'aatype': (4, 68), 'residue_index': (4, 68), 'seq_length': (4,), 'is_distillation': (4,), 'seq_mask': (4, 68), 'msa_mask': (4, 512, 68), 'msa_row_mask': (4, 512), 'random_crop_to_size_seed': (4, 2), 'atom14_atom_exists': (4, 68, 14), 'residx_atom14_to_atom37': (4, 68, 14), 'residx_atom37_to_atom14': (4, 68, 37), 'atom37_atom_exists': (4, 68, 37), 'extra_msa': (4, 5120, 68), 'extra_msa_mask': (4, 5120, 68), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 512, 68), 'true_msa': (4, 512, 68), 'extra_has_deletion': (4, 5120, 68), 'extra_deletion_value': (4, 5120, 68), 'msa_feat': (4, 512, 68, 49), 'target_feat': (4, 68, 22)}
I0127 12:01:47.091197 47607713561088 model.py:175] Output shape was {'distogram': {'bin_edges': (63,), 'logits': (68, 68, 64)}, 'experimentally_resolved': {'logits': (68, 37)}, 'masked_msa': {'logits': (512, 68, 23)}, 'predicted_lddt': {'logits': (68, 50)}, 'structure_module': {'final_atom_mask': (68, 37), 'final_atom_positions': (68, 37, 3)}, 'plddt': (68,), 'ranking_confidence': ()}
I0127 12:01:47.091758 47607713561088 run_alphafold.py:203] Total JAX model model_3_pred_0 on query predict time (includes compilation time, see --benchmark): 94.5s
I0127 12:01:50.797540 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {<Residue 67 (GLU) of chain 0>: ['OXT']}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:01:50.903615 47607713561088 amber_minimize.py:408] Minimizing protein, attempt 1 of 100.
I0127 12:01:51.603977 47607713561088 amber_minimize.py:69] Restraining 574 / 1170 particles.
I0127 12:01:53.237743 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:01:53.382962 47607713561088 amber_minimize.py:500] Iteration completed: Einit 1591.98 Efinal -2152.65 Time 1.38 s num residue violations 0 num residue exclusions 0
I0127 12:01:53.433610 47607713561088 run_alphafold.py:191] Running model model_4_pred_0 on query
I0127 12:01:54.805275 47607713561088 model.py:165] Running predict with shape(feat) = {'aatype': (4, 68), 'residue_index': (4, 68), 'seq_length': (4,), 'is_distillation': (4,), 'seq_mask': (4, 68), 'msa_mask': (4, 512, 68), 'msa_row_mask': (4, 512), 'random_crop_to_size_seed': (4, 2), 'atom14_atom_exists': (4, 68, 14), 'residx_atom14_to_atom37': (4, 68, 14), 'residx_atom37_to_atom14': (4, 68, 37), 'atom37_atom_exists': (4, 68, 37), 'extra_msa': (4, 5120, 68), 'extra_msa_mask': (4, 5120, 68), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 512, 68), 'true_msa': (4, 512, 68), 'extra_has_deletion': (4, 5120, 68), 'extra_deletion_value': (4, 5120, 68), 'msa_feat': (4, 512, 68, 49), 'target_feat': (4, 68, 22)}
I0127 12:03:17.109129 47607713561088 model.py:175] Output shape was {'distogram': {'bin_edges': (63,), 'logits': (68, 68, 64)}, 'experimentally_resolved': {'logits': (68, 37)}, 'masked_msa': {'logits': (512, 68, 23)}, 'predicted_lddt': {'logits': (68, 50)}, 'structure_module': {'final_atom_mask': (68, 37), 'final_atom_positions': (68, 37, 3)}, 'plddt': (68,), 'ranking_confidence': ()}
I0127 12:03:17.109581 47607713561088 run_alphafold.py:203] Total JAX model model_4_pred_0 on query predict time (includes compilation time, see --benchmark): 82.3s
I0127 12:03:20.810336 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {<Residue 67 (GLU) of chain 0>: ['OXT']}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:03:20.916513 47607713561088 amber_minimize.py:408] Minimizing protein, attempt 1 of 100.
I0127 12:03:21.116576 47607713561088 amber_minimize.py:69] Restraining 574 / 1170 particles.
I0127 12:03:22.792131 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:03:22.936354 47607713561088 amber_minimize.py:500] Iteration completed: Einit 1333.07 Efinal -2085.70 Time 0.96 s num residue violations 0 num residue exclusions 0
I0127 12:03:22.988237 47607713561088 run_alphafold.py:191] Running model model_5_pred_0 on query
I0127 12:03:24.791438 47607713561088 model.py:165] Running predict with shape(feat) = {'aatype': (4, 68), 'residue_index': (4, 68), 'seq_length': (4,), 'is_distillation': (4,), 'seq_mask': (4, 68), 'msa_mask': (4, 512, 68), 'msa_row_mask': (4, 512), 'random_crop_to_size_seed': (4, 2), 'atom14_atom_exists': (4, 68, 14), 'residx_atom14_to_atom37': (4, 68, 14), 'residx_atom37_to_atom14': (4, 68, 37), 'atom37_atom_exists': (4, 68, 37), 'extra_msa': (4, 1024, 68), 'extra_msa_mask': (4, 1024, 68), 'extra_msa_row_mask': (4, 1024), 'bert_mask': (4, 512, 68), 'true_msa': (4, 512, 68), 'extra_has_deletion': (4, 1024, 68), 'extra_deletion_value': (4, 1024, 68), 'msa_feat': (4, 512, 68, 49), 'target_feat': (4, 68, 22)}
I0127 12:04:56.475114 47607713561088 model.py:175] Output shape was {'distogram': {'bin_edges': (63,), 'logits': (68, 68, 64)}, 'experimentally_resolved': {'logits': (68, 37)}, 'masked_msa': {'logits': (512, 68, 23)}, 'predicted_lddt': {'logits': (68, 50)}, 'structure_module': {'final_atom_mask': (68, 37), 'final_atom_positions': (68, 37, 3)}, 'plddt': (68,), 'ranking_confidence': ()}
I0127 12:04:56.475727 47607713561088 run_alphafold.py:203] Total JAX model model_5_pred_0 on query predict time (includes compilation time, see --benchmark): 91.7s
I0127 12:05:00.706408 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {<Residue 67 (GLU) of chain 0>: ['OXT']}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:05:00.817349 47607713561088 amber_minimize.py:408] Minimizing protein, attempt 1 of 100.
I0127 12:05:01.640055 47607713561088 amber_minimize.py:69] Restraining 574 / 1170 particles.
I0127 12:05:03.474953 47607713561088 amber_minimize.py:178] alterations info: {'nonstandard_residues': [], 'removed_heterogens': set(), 'missing_residues': {}, 'missing_heavy_atoms': {}, 'missing_terminals': {}, 'Se_in_MET': [], 'removed_chains': {0: []}}
I0127 12:05:03.623641 47607713561088 amber_minimize.py:500] Iteration completed: Einit 1868.49 Efinal -2117.62 Time 1.63 s num residue violations 0 num residue exclusions 0
I0127 12:05:03.693668 47607713561088 run_alphafold.py:277] Final timings for query: {'features': 991.8286077976227, 'process_features_model_1_pred_0': 2.815833806991577, 'predict_and_compile_model_1_pred_0': 121.5330913066864, 'relax_model_1_pred_0': 14.062286376953125, 'process_features_model_2_pred_0': 1.3792037963867188, 'predict_and_compile_model_2_pred_0': 117.51480436325073, 'relax_model_2_pred_0': 5.612551212310791, 'process_features_model_3_pred_0': 1.1915929317474365, 'predict_and_compile_model_3_pred_0': 94.48047041893005, 'relax_model_3_pred_0': 5.277076721191406, 'process_features_model_4_pred_0': 1.3709683418273926, 'predict_and_compile_model_4_pred_0': 82.30484223365784, 'relax_model_4_pred_0': 4.778950452804565, 'process_features_model_5_pred_0': 1.802929162979126, 'predict_and_compile_model_5_pred_0': 91.68440127372742, 'relax_model_5_pred_0': 5.84104323387146}
ponomarevsy commented 1 year ago

more run_locally.bash

#!/bin/bash

#module load alphafold/2.3.1-Python-3.8.0
date=`printf '%(%Y-%m-%d)T\n' -1`
my_gpus=`/usr/local/bin/find_my_gpus.pl`

run_alphafold.sh -d /path/to/alphafold database -o ./dummy_test -m monomer -f /path/to/alphafold/2.3.1-Python-3.8.0/example/query.fasta -t $date -a $my_gpus