yaledeus / ElliDock

Rigid protein-protein docking via equivariant elliptic-paraboloid interface prediction
MIT License
2 stars 1 forks source link

Inference Fails #5

Closed ELMIAR-0642 closed 5 months ago

ELMIAR-0642 commented 5 months ago

Had to manually build the conda enviroinment and install the necessary libraries and packages, because clearly the python version mentioned in line 130@enviroinment.yml makes a conflict while installing so either creation freezes or fails.

Tried Checking out every step mentioned in the README.md

Can't Download and Split SabDab Data as mentioned here

Training code doesn't work. Gave the GPU id as 0 after verifying through the following code snippet,

import torch

# Check if GPU is available
if torch.cuda.is_available():
    # Get the GPU device ID
    gpu_id = torch.cuda.current_device()
    print(f"Using GPU with ID {gpu_id}")
else:
    print("GPU is not available. Using CPU.")

still it fails.

Moved on and tried to run inference code, and it gave Segmentation fault (core dumped) error. Tried to trace it back and added try-exception block above every function calls, and print statements, and found the line model = torch.load(args.ckpt, map_location='cpu'), in test.py does not load the model file, for some reason.

Output after adding Try-Except blocks can be seen here:

python test.py --dataset DB5 --gpu 0 --ckpt ./checkpoints/db5_best.ckpt

Model type: ElliDock DB5 1706607943.9382849 ./checkpoints/db5_best_results

Can you please Address the issue as early as possible?

Segmentation fault (core dumped)

yaledeus commented 5 months ago

For the environment

Sorry for the inconvenience caused by the incompatible environment files environment.yml. We have updated README and list external python packages that need to be downloaded.

For dataset

The bash file works well in my server with the following outputs:

PDB_DIR: ***
download sabdab from summary file ../summaries/sabdab_summary_all.tsv
using local PDB files: **/all_structures/imgt
PDB file already renumbered with scheme imgt
downloading raw files
8CAF not found in **/all_structures/imgt, try fetching from remote server
7XY4 not found in **/all_structures/imgt, try fetching from remote server
7XY3 not found in **/all_structures/imgt, try fetching from remote server
7Y0N not found in **/all_structures/imgt, try fetching from remote server
0%|                                                                                            | 0/7709 [00:00<?, ?it/s]

It should be noted that the downloaded items are loaded in memory until the downloading process finishes. We recommend a more efficient way mentioned here. First download the zip file from the official website of SAbDab as shown below:

image

Then you can put the unzipped directory under your PDB_DIR and run download_and_split.sh.

For training and evaluation

From your description, I'm not quite sure what your specific problem is during the training phase. Similarly, we cannot reproduce the error you reported during the inference phase. Part of the outputs of running the inference code on our machine are as follows: ​python test.py --dataset DB5 --gpu 0 --ckpt ./checkpoints/db5_best.ckpt

Model type: ElliDock 100%████████████████████████████████| 25/25 [00:55<00:00, 2.22s/it] total runtime: 81.07781410217285 CRMSD(aligned) median: 12.995396614074707 mean: 14.41262475013733 std: 6.779665588791658 IRMSD(aligned) median: 11.134011268615723 mean: 12.47975996017456 std: 4.965853350631719

Please confirm whether your environment contains incompatible packages or other related problems.

ELMIAR-0642 commented 5 months ago

@yaledeus Thanks. I am seeing your comment now only. I tried and found the issue with model loading failure is a torch version mismatch. I fixed it after reading here, and a better way to create an ellidock environment that I feel is to create it and manually install the packages as shown below,

conda create --name ellidock python==3.8
conda activate ellidock
pip install BioPython tqdm && \
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge  && \
pip install torch-cluster torch-scatter --extra-index-url https://data.pyg.org/whl/torch-1.10.0%2Bcu113.html && \
pip install pandas tensorboard==2.12.1 e3nn==0.5.1

But I will read the new updated README.md.

So, Now, while inferencing, the script loads the model, but I got the following error:

python test.py --dataset DB5 --gpu 0 --ckpt ./checkpoints/db5_best.ckpt

Model type: ElliDock DB5 1706673281.5021484 ./checkpoints/db5_best_results ./checkpoints/db5_best.ckpt cuda:0 ElliDock( (graph_constructor): ComplexGraph( (aa_embedding): AminoAcidEmbedding( (residue_embedding): Embedding(20, 64) (res_pos_embedding): SinusoidalPositionEmbedding() ) ) (in_conv): Sequential( (0): Linear(in_features=67, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gnn_0): PINN( (PI_MP_Block): Polarizable_Interaction_MP_Block( (node_conv): Sequential( (0): Linear(in_features=160, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) (rbf_conv): Sequential( (0): Linear(in_features=20, out_features=384, bias=True) (1): Dropout(p=0.1, inplace=False) ) (transformer_gcl): Transformer_GCL( (linear_q0): Linear(in_features=384, out_features=384, bias=False) (linear_k0): Linear(in_features=384, out_features=384, bias=False) (linear_v0): Linear(in_features=384, out_features=384, bias=False) (linear_q1): Linear(in_features=384, out_features=384, bias=False) (linear_k1): Linear(in_features=384, out_features=384, bias=False) (linear_v1): Linear(in_features=384, out_features=384, bias=False) (linear_q2): Linear(in_features=384, out_features=384, bias=False) (linear_k2): Linear(in_features=384, out_features=384, bias=False) (linear_v2): Linear(in_features=384, out_features=384, bias=False) (linear_q3): Linear(in_features=384, out_features=384, bias=False) (linear_k3): Linear(in_features=384, out_features=384, bias=False) (linear_v3): Linear(in_features=384, out_features=384, bias=False) (linear_out): Sequential( (0): Linear(in_features=384, out_features=384, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=384, out_features=384, bias=True) ) ) ) (PI_update_Block): Polarizable_Interaction_Update_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) ) ) (inter_act_0): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gnn_1): PINN( (PI_MP_Block): Polarizable_Interaction_MP_Block( (node_conv): Sequential( (0): Linear(in_features=160, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) (rbf_conv): Sequential( (0): Linear(in_features=20, out_features=384, bias=True) (1): Dropout(p=0.1, inplace=False) ) (transformer_gcl): Transformer_GCL( (linear_q0): Linear(in_features=384, out_features=384, bias=False) (linear_k0): Linear(in_features=384, out_features=384, bias=False) (linear_v0): Linear(in_features=384, out_features=384, bias=False) (linear_q1): Linear(in_features=384, out_features=384, bias=False) (linear_k1): Linear(in_features=384, out_features=384, bias=False) (linear_v1): Linear(in_features=384, out_features=384, bias=False) (linear_q2): Linear(in_features=384, out_features=384, bias=False) (linear_k2): Linear(in_features=384, out_features=384, bias=False) (linear_v2): Linear(in_features=384, out_features=384, bias=False) (linear_q3): Linear(in_features=384, out_features=384, bias=False) (linear_k3): Linear(in_features=384, out_features=384, bias=False) (linear_v3): Linear(in_features=384, out_features=384, bias=False) (linear_out): Sequential( (0): Linear(in_features=384, out_features=384, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=384, out_features=384, bias=True) ) ) ) (PI_update_Block): Polarizable_Interaction_Update_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) ) ) (inter_act_1): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gnn_2): PINN( (PI_MP_Block): Polarizable_Interaction_MP_Block( (node_conv): Sequential( (0): Linear(in_features=160, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) (rbf_conv): Sequential( (0): Linear(in_features=20, out_features=384, bias=True) (1): Dropout(p=0.1, inplace=False) ) (transformer_gcl): Transformer_GCL( (linear_q0): Linear(in_features=384, out_features=384, bias=False) (linear_k0): Linear(in_features=384, out_features=384, bias=False) (linear_v0): Linear(in_features=384, out_features=384, bias=False) (linear_q1): Linear(in_features=384, out_features=384, bias=False) (linear_k1): Linear(in_features=384, out_features=384, bias=False) (linear_v1): Linear(in_features=384, out_features=384, bias=False) (linear_q2): Linear(in_features=384, out_features=384, bias=False) (linear_k2): Linear(in_features=384, out_features=384, bias=False) (linear_v2): Linear(in_features=384, out_features=384, bias=False) (linear_q3): Linear(in_features=384, out_features=384, bias=False) (linear_k3): Linear(in_features=384, out_features=384, bias=False) (linear_v3): Linear(in_features=384, out_features=384, bias=False) (linear_out): Sequential( (0): Linear(in_features=384, out_features=384, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=384, out_features=384, bias=True) ) ) ) (PI_update_Block): Polarizable_Interaction_Update_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=384, bias=True) ) ) ) (inter_act_2): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=128, bias=True) ) (gated_equiv_block): Gated_Equivariant_Block( (vec_u_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (vec_v_fc): Sequential( (0): Linear(in_features=128, out_features=128, bias=False) (1): Dropout(p=0.1, inplace=False) ) (node_conv): Sequential( (0): Linear(in_features=256, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=256, bias=True) ) ) (re_inv_conv): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=4, bias=True) ) (li_inv_conv): Sequential( (0): Linear(in_features=128, out_features=128, bias=True) (1): SiLU() (2): Dropout(p=0.1, inplace=False) (3): Linear(in_features=128, out_features=4, bias=True) ) (re_equiv_conv): Sequential( (0): Linear(in_features=128, out_features=10, bias=False) (1): Dropout(p=0.1, inplace=False) ) (li_equiv_conv): Sequential( (0): Linear(in_features=128, out_features=10, bias=False) (1): Dropout(p=0.1, inplace=False) ) (neg_penalty_loss): NegPenaltyLoss() (out_span_loss): OutSpanLoss() (normalizer): CoordNomralizer() ) 0%| | 0/25 [00:00<?, ?it/s]1 2 0%| | 0/25 [00:00<?, ?it/s] Traceback (most recent call last): File "test.py", line 295, in main(parse()) File "test.py", line 137, in main dock_X, dock_trans_list = model.dock(batch) # (N, 3) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/model.py", line 301, in dock intra_node_feat, vec_feat = self.modules[f'gnn{i}']( File "/opt/conda/envs/ellidock2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/gnn.py", line 454, in forward m_node_agg, m_vec_agg = self.PI_MP_Block( File "/opt/conda/envs/ellidock2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/gnn.py", line 381, in forward node_out = self.transformer_gcl(node_out, edges) # (n_edges, 3 hidden_nf) File "/opt/conda/envs/ellidock2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, **kwargs) File "/home/ubuntu/BoltPro/January2024/Experiments/New_Experiments/Docking/ElliDock/module/gnn.py", line 274, in forward Att = scatter_softmax(Att, row).unsqueeze(1) # (n_edges, 1) File "/home/ubuntu/.local/lib/python3.8/site-packages/torch_scatter/composite/softmax.py", line 18, in scatter_softmax max_value_per_index = scatter_max( File "/home/ubuntu/.local/lib/python3.8/site-packages/torch_scatter/scatter.py", line 72, in scatter_max return torch.ops.torch_scatter.scatter_max(src, index, dim, out, dim_size) RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

yaledeus commented 5 months ago

Thank you for your advice for constructing the environment. From the error output below, I guess you have downloaded torch_scatter that does not match your CUDA version, please check your library or use only cpu, i.e. do not specify --gpu.

image
yaledeus commented 5 months ago

Well, this is due to the fact that you haven't compiled TMscore into an executable file, as I mentioned in README:

image
ELMIAR-0642 commented 5 months ago

Fine I have installed TMscore and DockQ has been installed along with EMBOSS-6.x.x/needle. Thanks for the help.

Still have not downloaded the SabDab.

(ellidock2) root@b41982095b31:/ElliDock# python test.py --dataset DB5 --gpu 0 --ckpt ./checkpoints/db5_best.ckpt

Model type: ElliDock 100%|█████████████████████████████████████████████████████████████████████████████████████| 25/25 [04:19<00:00, 10.36s/it] total runtime: 265.49426341056824 CRMSD(aligned) median: 12.995396614074707 mean: 14.412623949050904 std: 6.779664219326272 IRMSD(aligned) median: 11.134014129638672 mean: 12.479758749008178 std: 4.9658500449823375 TMscore median: 1.0 mean: 1.0 std: 0.0 DockQ median: 0.037 mean: 0.059960000000000006 std: 0.06012618730636427 CRMSD median: 33.692474365234375 mean: 35.48002140045166 std: 15.150973126984095 IRMSD median: 17.320070266723633 mean: 18.748350467681885 std: 7.188218830565851

Closing this issue.