Closed ELMIAR-0642 closed 9 months ago
Sorry for the inconvenience caused by the incompatible environment files environment.yml
. We have updated README and list external python packages that need to be downloaded.
The bash file works well in my server with the following outputs:
PDB_DIR: ***
download sabdab from summary file ../summaries/sabdab_summary_all.tsv
using local PDB files: **/all_structures/imgt
PDB file already renumbered with scheme imgt
downloading raw files
8CAF not found in **/all_structures/imgt, try fetching from remote server
7XY4 not found in **/all_structures/imgt, try fetching from remote server
7XY3 not found in **/all_structures/imgt, try fetching from remote server
7Y0N not found in **/all_structures/imgt, try fetching from remote server
0%| | 0/7709 [00:00<?, ?it/s]
It should be noted that the downloaded items are loaded in memory until the downloading process finishes. We recommend a more efficient way mentioned here. First download the zip file from the official website of SAbDab as shown below:
Then you can put the unzipped directory under your PDB_DIR
and run download_and_split.sh
.
From your description, I'm not quite sure what your specific problem is during the training phase. Similarly, we cannot reproduce the error you reported during the inference phase. Part of the outputs of running the inference code on our machine are as follows:
python test.py --dataset DB5 --gpu 0 --ckpt ./checkpoints/db5_best.ckpt
Model type: ElliDock 100%████████████████████████████████| 25/25 [00:55<00:00, 2.22s/it] total runtime: 81.07781410217285 CRMSD(aligned) median: 12.995396614074707 mean: 14.41262475013733 std: 6.779665588791658 IRMSD(aligned) median: 11.134011268615723 mean: 12.47975996017456 std: 4.965853350631719
Please confirm whether your environment contains incompatible packages or other related problems.
@yaledeus Thanks. I am seeing your comment now only. I tried and found the issue with model loading failure is a torch version mismatch. I fixed it after reading here, and a better way to create an ellidock environment that I feel is to create it and manually install the packages as shown below,
conda create --name ellidock python==3.8
conda activate ellidock
pip install BioPython tqdm && \
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge && \
pip install torch-cluster torch-scatter --extra-index-url https://data.pyg.org/whl/torch-1.10.0%2Bcu113.html && \
pip install pandas tensorboard==2.12.1 e3nn==0.5.1
But I will read the new updated README.md
.
So, Now, while inferencing, the script loads the model, but I got the following error:
python test.py --dataset DB5 --gpu 0 --ckpt ./checkpoints/db5_best.ckpt
Model type: ElliDock DB5 1706673281.5021484 ./checkpoints/db5_best_results ./checkpoints/db5_best.ckpt cuda:0 ElliDock( (graph_constructor): ComplexGraph( (aa_embedding): AminoAcidEmbedding( (residue_embedding): Embedding(20, 64) (res_pos_embedding): SinusoidalPositionEmbedding() ) ) (in_conv): Sequential( (0): Linear(in_features=67, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gnn_0): PINN( (PI_MP_Block): Polarizable_Interaction_MP_Block( (node_conv): Sequential( (0): Linear(in_features=160, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) (rbf_conv): Sequential( (0): Linear(in_features=20, out_features=384, bias=True) (1): Dropout(p=0.1, inplace=False) ) (transformer_gcl): Transformer_GCL( (linear_q0): Linear(in_features=384, out_features=384, bias=False) (linear_k0): Linear(in_features=384, out_features=384, bias=False) (linear_v0): Linear(in_features=384, out_features=384, bias=False) (linear_q1): Linear(in_features=384, out_features=384, bias=False) (linear_k1): Linear(in_features=384, out_features=384, bias=False) (linear_v1): Linear(in_features=384, out_features=384, bias=False) (linear_q2): Linear(in_features=384, out_features=384, bias=False) (linear_k2): Linear(in_features=384, out_features=384, bias=False) (linear_v2): Linear(in_features=384, out_features=384, bias=False) (linear_q3): Linear(in_features=384, out_features=384, bias=False) (linear_k3): Linear(in_features=384, out_features=384, bias=False) (linear_v3): Linear(in_features=384, out_features=384, bias=False) (linear_out): Sequential( (0): Linear(in_features=384, out_features=384, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=384, out_features=384, bias=True) ) ) ) (PI_update_Block): Polarizable_Interaction_Update_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) ) ) (inter_act_0): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gnn_1): PINN( (PI_MP_Block): Polarizable_Interaction_MP_Block( (node_conv): Sequential( (0): Linear(in_features=160, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) (rbf_conv): Sequential( (0): Linear(in_features=20, out_features=384, bias=True) (1): Dropout(p=0.1, inplace=False) ) (transformer_gcl): Transformer_GCL( (linear_q0): Linear(in_features=384, out_features=384, bias=False) (linear_k0): Linear(in_features=384, out_features=384, bias=False) (linear_v0): Linear(in_features=384, out_features=384, bias=False) (linear_q1): Linear(in_features=384, out_features=384, bias=False) (linear_k1): Linear(in_features=384, out_features=384, bias=False) (linear_v1): Linear(in_features=384, out_features=384, bias=False) (linear_q2): Linear(in_features=384, out_features=384, bias=False) (linear_k2): Linear(in_features=384, out_features=384, bias=False) (linear_v2): Linear(in_features=384, out_features=384, bias=False) (linear_q3): Linear(in_features=384, out_features=384, bias=False) (linear_k3): Linear(in_features=384, out_features=384, bias=False) (linear_v3): Linear(in_features=384, out_features=384, bias=False) (linear_out): Sequential( (0): Linear(in_features=384, out_features=384, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=384, out_features=384, bias=True) ) ) ) (PI_update_Block): Polarizable_Interaction_Update_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) ) ) (inter_act_1): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gnn_2): PINN( (PI_MP_Block): Polarizable_Interaction_MP_Block( (node_conv): Sequential( (0): Linear(in_features=160, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) (rbf_conv): Sequential( (0): Linear(in_features=20, out_features=384, bias=True) (1): Dropout(p=0.1, inplace=False) ) (transformer_gcl): Transformer_GCL( (linear_q0): Linear(in_features=384, out_features=384, bias=False) (linear_k0): Linear(in_features=384, out_features=384, bias=False) (linear_v0): Linear(in_features=384, out_features=384, bias=False) (linear_q1): Linear(in_features=384, out_features=384, bias=False) (linear_k1): Linear(in_features=384, out_features=384, bias=False) (linear_v1): Linear(in_features=384, out_features=384, bias=False) (linear_q2): Linear(in_features=384, out_features=384, bias=False) (linear_k2): Linear(in_features=384, out_features=384, bias=False) (linear_v2): Linear(in_features=384, out_features=384, bias=False) (linear_q3): Linear(in_features=384, out_features=384, bias=False) (linear_k3): Linear(in_features=384, out_features=384, bias=False) (linear_v3): Linear(in_features=384, out_features=384, bias=False) (linear_out): Sequential( (0): Linear(in_features=384, out_features=384, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=384, out_features=384, bias=True) ) ) ) (PI_update_Block): Polarizable_Interaction_Update_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) ) ) (inter_act_2): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gated_equiv_block): Gated_Equivariant_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=256, bias=True) ) ) (re_inv_conv): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=4, bias=True) ) (li_inv_conv): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=4, bias=True) ) (re_equiv_conv): Sequential( (0): Linear(in_features=128, out_features=10, bias=False) (1): Dropout(p=0.1, inplace=False) ) (li_equiv_conv): Sequential( (0): Linear(in_features=128, out_features=10, bias=False) (1): Dropout(p=0.1, inplace=False) ) (neg_penalty_loss): NegPenaltyLoss() (out_span_loss): OutSpanLoss() (normalizer): CoordNomralizer() ) 0%| | 0/25 [00:00<?, ?it/s]1 2 0%| | 0/25 [00:00<?, ?it/s] Traceback (most recent call last): File "test.py", line 295, in
main(parse()) File "test.py", line 137, in main dock_X, dock_trans_list = model.dock(batch) # (N, 3) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/model.py", line 301, in dock intra_node_feat, vec_feat = self.modules[f'gnn{i}']( File "/opt/conda/envs/ellidock2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/gnn.py", line 454, in forward m_node_agg, m_vec_agg = self.PI_MP_Block( File "/opt/conda/envs/ellidock2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/gnn.py", line 381, in forward node_out = self.transformer_gcl(node_out, edges) # (n_edges, 3 hidden_nf) File "/opt/conda/envs/ellidock2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, **kwargs) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/gnn.py", line 274, in forward Att = scatter_softmax(Att, row).unsqueeze(1) # (n_edges, 1) File "/home/ubuntu/.local/lib/python3.8/site-packages/torch_scatter/composite/softmax.py", line 18, in scatter_softmax max_value_per_index = scatter_max( File "/home/ubuntu/.local/lib/python3.8/site-packages/torch_scatter/scatter.py", line 72, in scatter_max return torch.ops.torch_scatter.scatter_max(src, index, dim, out, dim_size) RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Thank you for your advice for constructing the environment.
From the error output below, I guess you have downloaded torch_scatter
that does not match your CUDA version, please check your library or use only cpu, i.e. do not specify --gpu
.
Well, this is due to the fact that you haven't compiled TMscore into an executable file, as I mentioned in README:
Fine I have installed TMscore and DockQ has been installed along with EMBOSS-6.x.x/needle. Thanks for the help.
Still have not downloaded the SabDab.
(ellidock2) root@b41982095b31:/ElliDock# python test.py --dataset DB5 --gpu 0 --ckpt ./checkpoints/db5_best.ckpt
Model type: ElliDock 100%|█████████████████████████████████████████████████████████████████████████████████████| 25/25 [04:19<00:00, 10.36s/it] total runtime: 265.49426341056824 CRMSD(aligned) median: 12.995396614074707 mean: 14.412623949050904 std: 6.779664219326272 IRMSD(aligned) median: 11.134014129638672 mean: 12.479758749008178 std: 4.9658500449823375 TMscore median: 1.0 mean: 1.0 std: 0.0 DockQ median: 0.037 mean: 0.059960000000000006 std: 0.06012618730636427 CRMSD median: 33.692474365234375 mean: 35.48002140045166 std: 15.150973126984095 IRMSD median: 17.320070266723633 mean: 18.748350467681885 std: 7.188218830565851
Closing this issue.
Had to manually build the conda enviroinment and install the necessary libraries and packages, because clearly the python version mentioned in line 130@enviroinment.yml makes a conflict while installing so either creation freezes or fails.
Tried Checking out every step mentioned in the README.md
Can't Download and Split SabDab Data as mentioned here
Training code doesn't work. Gave the GPU id as 0 after verifying through the following code snippet,
still it fails.
Moved on and tried to run inference code, and it gave
Segmentation fault (core dumped)
error. Tried to trace it back and added try-exception block above every function calls, and print statements, and found the linemodel = torch.load(args.ckpt, map_location='cpu')
, intest.py
does not load the model file, for some reason.Output after adding Try-Except blocks can be seen here:
Can you please Address the issue as early as possible?
Segmentation fault (core dumped)