search

uw-ipd / RoseTTAFold2NA

RoseTTAFold2 protein/nucleic acid complex prediction
MIT License
310 stars 69 forks source link

Get errors "NVTX functions not installed. Are you sure you have a CUDA build?" when run RF2NA on CPU. #36

Open Wenhao-Jin opened 1 year ago

Wenhao-Jin commented 1 year ago

Hi,

I get the following errors when run RoseTTAFold2NA on CPU. I have replaced the "torch.cuda.amp.autocast" with "torch.amp.autocast" in predict.py in order to overcome another "NVTX functions not installed." error when running the run_RF2NA.sh.

Seems there is some part of the codes still calling cuda or searching for GPUs? Thank you!

Traceback (most recent call last):
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/predict.py", line 376, in <module>
    pred.predict(inputs=args.inputs, out_prefix=args.prefix, ffdb=ffdb)
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/predict.py", line 239, in predict
    self._run_model(Ls, msa_orig, ins_orig, t1d, t2d, xyz_t, xyz_t[:,0], alpha_t, "%s_%02d"%(out_prefix, i_trial))
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/predict.py", line 299, in _run_model
    logit_s, logit_aa_s, logit_pae, init_crds, alpha_prev, _, pred_lddt_binned, msa_prev, pair_prev, state_prev = self.model(
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/RoseTTAFoldModel.py", line 104, in forward
    msa, pair, xyz, alpha_s, xyzallatom, state = self.simulator(
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/Track_module.py", line 441, in forward
    msa_full, pair, xyz, state, alpha = self.extra_block[i_m](msa_full, pair,
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/Track_module.py", line 367, in forward
    xyz, state, alpha = self.str2str(msa.float(), pair.float(), xyz.detach().float(), state.float(), idx, top_k=top_k)
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/Track_module.py", line 234, in forward
    shift = self.se3(G, node.reshape(B*L, -1, 1), l1_feats, edge_feats)
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/expanse/lustre/projects/ddp398/wjin/software/RoseTTAFold2NA/network/SE3_network.py", line 84, in forward
    return self.se3(G, node_features, edge_features) #, clamp_d=clamp_d)
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/se3_transformer-1.0.0-py3.8.egg/se3_transformer/model/transformer.py", line 163, in forward
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/se3_transformer-1.0.0-py3.8.egg/se3_transformer/model/basis.py", line 166, in get_basis
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/contextlib.py", line 114, in __enter__
    return next(self.gen)
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/cuda/nvtx.py", line 86, in range
    range_push(msg.format(*args, **kwargs))
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/cuda/nvtx.py", line 28, in range_push
    return _nvtx.rangePushA(msg)
  File "/home/wjin/data/anaconda3/envs/RF2NA/lib/python3.8/site-packages/torch/cuda/nvtx.py", line 9, in _fail
    raise RuntimeError("NVTX functions not installed. Are you sure you have a CUDA build?")
RuntimeError: NVTX functions not installed. Are you sure you have a CUDA build?

My package versions:

brotlipy==0.7.0
certifi @ file:///croot/certifi_1665076670883/work/certifi
cffi @ file:///tmp/abs_98z5h56wf8/croots/recipe/cffi_1659598650955/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click==8.1.3
colorama @ file:///opt/conda/conda-bld/colorama_1657009087971/work
configparser==5.3.0
cryptography @ file:///croot/cryptography_1665612644927/work
dgl==0.9.1.post1
DLLogger @ git+https://github.com/NVIDIA/dllogger@0540a43971f4a8a16693a9de9de73c1072020769
docker-pycreds==0.4.0
e3nn==0.3.3
gitdb==4.0.9
GitPython==3.1.29
idna @ file:///croot/idna_1666125576474/work
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186064646/work
mkl-service==2.4.0
mpmath==1.2.1
networkx @ file:///opt/conda/conda-bld/networkx_1657784097507/work
numpy @ file:///croot/numpy_and_numpy_base_1667233465264/work
opt-einsum==3.3.0
opt-einsum-fx==0.1.4
packaging==21.3
pathtools==0.1.2
promise==2.3
protobuf==4.21.9
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1667885878918/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pynvml==11.0.0
pyOpenSSL @ file:///opt/conda/conda-bld/pyopenssl_1643788558760/work
pyparsing==3.0.9
PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work
python-dateutil==2.8.2
PyYAML==6.0
requests @ file:///opt/conda/conda-bld/requests_1657734628632/work
scipy==1.9.3
se3-transformer==1.0.0
sentry-sdk==1.11.0
shortuuid==1.0.11
six @ file:///tmp/build/80754af9/six_1644875935023/work
smmap==5.0.0
subprocess32==3.5.4
sympy==1.11.1
torch==1.13.0
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1662214488106/work
typing_extensions @ file:///tmp/abs_ben9emwtky/croots/recipe/typing_extensions_1659638822008/work
urllib3 @ file:///croot/urllib3_1666298941550/work
wandb==0.12.0
robindbugent commented 1 year ago

I am having the same issue. Any updates on this?

kcygan commented 1 year ago

The environment file is missing a dependency for pytorch to see cuda. You have to add/modify lines in the environment file: Specify - pytorch::pytorch=1.12.0 and - pytorch::torchvision=0.13.0 and the issue is going to go away.

Here is my modified conda environment file: 

cat > RF2.yaml <<EOF
name: RF2NA
channels:
  - defaults
dependencies:
  - python=3.8
  - pytorch::pytorch=1.12.0
  - pytorch::torchvision=0.13.0
  - requests
  - conda-forge::psutil
  - conda-forge::cudatoolkit=11.3
  - conda-forge::tqdm
  - dglteam::dgl-cuda11.3
  - bioconda::mafft
  - bioconda::hhsuite
  - bioconda::blast
  - bioconda::hmmer>=3.3
  - bioconda::infernal
  - bioconda::cd-hit
  - bioconda::csblast
  - biocore::psipred=4.01
  - biocore::blast-legacy=2.2.26
EOF

conda env create -f RF2.yaml

Cheers,

Kamil

Wenhao-Jin commented 1 year ago

Thanks @kcygan! The NVTX errors disappeared. But I got a new DGL error shown below.

The details are here:

  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/predict.py", line 376, in <module>
    pred.predict(inputs=args.inputs, out_prefix=args.prefix, ffdb=ffdb)
  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/predict.py", line 239, in predict
    self._run_model(Ls, msa_orig, ins_orig, t1d, t2d, xyz_t, xyz_t[:,0], alpha_t, "%s_%02d"%(out_prefix, i_trial))
  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/predict.py", line 299, in _run_model
    logit_s, logit_aa_s, logit_pae, init_crds, alpha_prev, _, pred_lddt_binned, msa_prev, pair_prev, state_prev = self.model(
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/RoseTTAFoldModel.py", line 104, in forward
    msa, pair, xyz, alpha_s, xyzallatom, state = self.simulator(
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/Track_module.py", line 441, in forward
    msa_full, pair, xyz, state, alpha = self.extra_block[i_m](msa_full, pair,
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/Track_module.py", line 367, in forward
    xyz, state, alpha = self.str2str(msa.float(), pair.float(), xyz.detach().float(), state.float(), idx, top_k=top_k)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
    return func(*args, **kwargs)
  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/Track_module.py", line 234, in forward
    shift = self.se3(G, node.reshape(B*L, -1, 1), l1_feats, edge_feats)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/MY_HOME_FOLDER/software/RoseTTAFold2NA/network/SE3_network.py", line 84, in forward
    return self.se3(G, node_features, edge_features) #, clamp_d=clamp_d)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/se3_transformer-1.0.0-py3.8.egg/se3_transformer/model/transformer.py", line 176, in forward
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/se3_transformer-1.0.0-py3.8.egg/se3_transformer/model/transformer.py", line 47, in forward
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/se3_transformer-1.0.0-py3.8.egg/se3_transformer/model/layers/attention.py", line 169, in forward
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/se3_transformer-1.0.0-py3.8.egg/se3_transformer/model/layers/attention.py", line 82, in forward
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/ops/sddmm.py", line 127, in func
    return gsddmm(g, binary_op, x, y,
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/ops/sddmm.py", line 74, in gsddmm
    return gsddmm_internal(
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/backend/pytorch/sparse.py", line 733, in gsddmm
    return GSDDMM.apply(gidx, op, lhs_data, rhs_data, lhs_target, rhs_target)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 118, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/backend/pytorch/sparse.py", line 289, in forward
    out = _gsddmm(gidx, op, X, Y, lhs_target, rhs_target)
  File "/home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/sparse.py", line 501, in _gsddmm
    _CAPI_DGLKernelSDDMM(gidx, op,
  File "dgl/_ffi/_cython/./function.pxi", line 293, in dgl._ffi._cy3.core.FunctionBase.__call__
  File "dgl/_ffi/_cython/./function.pxi", line 239, in dgl._ffi._cy3.core.FuncCall
dgl._ffi.base.DGLError: [14:01:27] /opt/dgl/src/array/kernel.cc:257: Check failed: (out->dtype).code == kDLFloat (^D vs. 2) : Feature data must be float type
Stack trace:
  [bt] (0) /home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/libdgl.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4f) [0x15549fbcd1cf]
  [bt] (1) /home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/libdgl.so(dgl::aten::SDDMM(std::string const&, std::shared_ptr<dgl::BaseHeteroGraph>, dgl::runtime::NDArray, dgl::runtime::NDArray, dgl::runtime::NDArray, int, int)+0x3f8) [0x15549fe89558]
  [bt] (2) /home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/libdgl.so(+0x66d40e) [0x15549fea140e]
  [bt] (3) /home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/libdgl.so(+0x66da21) [0x15549fea1a21]
  [bt] (4) /home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/libdgl.so(DGLFuncCall+0x48) [0x15549fef3708]
  [bt] (5) /home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/_ffi/_cy3/core.cpython-38-x86_64-linux-gnu.so(+0x16f93) [0x15549ed8ff93]
  [bt] (6) /home/wjin/data/anaconda3/envs/RF2NA_2/lib/python3.8/site-packages/dgl/_ffi/_cy3/core.cpython-38-x86_64-linux-gnu.so(+0x1777b) [0x15549ed9077b]
  [bt] (7) python(_PyObject_MakeTpCall+0x3eb) [0x4d207b]
  [bt] (8) python(_PyEval_EvalFrameDefault+0x4f68) [0x4cd0a8]

The current package versions:


certifi @ file:///croot/certifi_1671487769961/work/certifi
cffi @ file:///croot/cffi_1670423208954/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click==8.1.3
colorama @ file:///croot/colorama_1672386526460/work
configparser==5.3.0
cryptography @ file:///croot/cryptography_1673298753778/work
dgl==0.9.1.post1
DLLogger @ git+https://github.com/NVIDIA/dllogger@0540a43971f4a8a16693a9de9de73c1072020769
docker-pycreds==0.4.0
e3nn==0.3.3
flit_core @ file:///opt/conda/conda-bld/flit-core_1644941570762/work/source/flit_core
gitdb==4.0.10
GitPython==3.1.30
idna @ file:///croot/idna_1666125576474/work
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186064646/work
mkl-service==2.4.0
mpmath==1.2.1
networkx @ file:///opt/conda/conda-bld/networkx_1657784097507/work
numpy @ file:///croot/numpy_and_numpy_base_1672336185480/work
opt-einsum==3.3.0
opt-einsum-fx==0.1.4
packaging==23.0
pathtools==0.1.2
Pillow==9.3.0
promise==2.3
protobuf==4.21.12
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1667885878918/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pynvml==11.0.0
pyOpenSSL @ file:///opt/conda/conda-bld/pyopenssl_1643788558760/work
PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work
python-dateutil==2.8.2
PyYAML==6.0
requests @ file:///opt/conda/conda-bld/requests_1657734628632/work
scipy==1.9.3
se3-transformer==1.0.0
sentry-sdk==1.14.0
shortuuid==1.0.11
six @ file:///tmp/build/80754af9/six_1644875935023/work
smmap==5.0.0
subprocess32==3.5.4
sympy==1.11.1
torch==1.12.0
torchvision==0.13.0
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1662214488106/work
typing_extensions @ file:///croot/typing_extensions_1669924550328/work
urllib3 @ file:///croot/urllib3_1673575502006/work
wandb==0.12.0```
DNARNAi commented 1 year ago

Hiii I have reported same issue I have updated the software as you mentioned but the problem still there can't figure out what to do.. This problem I have faced in HPC installation.... Help me out what to do...

caseyh9438 commented 1 year ago

I resolved this issue by uninstalling Pytorch and reinstalling Pytorch with cuda support conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia

v-shaoningli commented 9 months ago

run pip install dgl -f https://data.dgl.ai/wheels/cu118/repo.html

fglaser commented 8 months ago

Dear authors,

I have found the same problem, tried caseyh9438 and v-shaoningli suggestions but still having exactly the same error:

I paste the full output just in case somebody can advise, Thanks a lot! Fabian

(RF2NA) rosetta@amber3:~/RoseTTAFold2NA/example$ ../run_RF2NA.sh rna_pred rna_binding_protein.fa R:RNA.fa Creating joint Protein/RNA MSA -> Running command: /home/rosetta/RoseTTAFold2NA/input_prep/make_rna_msa.sh RNA.fa /home/rosetta/RoseTTAFold2NA/example/rna_pred RNA 8 64 Running RoseTTAFold2NA to predict structures -> Running command: python /home/rosetta/RoseTTAFold2NA/network/predict.py -inputs PR:/home/rosetta/RoseTTAFold2NA/example/rna_pred/rna_binding_protein.RNA.a3m:/home/rosetta/RoseTTAFold2NA/example/rna_pred/rna_binding_protein.hhr:/home/rosetta/RoseTTAFold2NA/example/rna_pred/rna_binding_protein.atab -prefix /home/rosetta/RoseTTAFold2NA/example/rna_pred/models/model -model /home/rosetta/RoseTTAFold2NA/network/weights/RF2NA_apr23.pt -db /home/rosetta/RoseTTAFold2NA/pdb100_2021Mar03/pdb100_2021Mar03 Running on CPU plddt best Traceback (most recent call last): File "/home/rosetta/RoseTTAFold2NA/network/predict.py", line 374, in pred.predict(inputs=args.inputs, out_prefix=args.prefix, ffdb=ffdb) File "/home/rosetta/RoseTTAFold2NA/network/predict.py", line 250, in predict self._run_model(Ls, msa_orig, ins_orig, t1d, t2d, xyz_t, xyz_t[:,0], alpha_t, same_chain, mask_t2d, "%s%02d"%(out_prefix, i_trial)) File "/home/rosetta/RoseTTAFold2NA/network/predict.py", line 296, in _run_model logit_s, logit_aa_s, logit_pae, p_bind, init_crds, alphaprev, , pred_lddt_binned, msa_prev, pair_prev, state_prev = self.model( File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/rosetta/RoseTTAFold2NA/network/RoseTTAFoldModel.py", line 91, in forward msa, pair, xyz, alpha, xyzallatom, state = self.simulator( File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/rosetta/RoseTTAFold2NA/network/Track_module.py", line 465, in forward msa_full, pair, xyz, state, alpha = self.extra_block[i_m](msa_full, pair, File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/rosetta/RoseTTAFold2NA/network/Track_module.py", line 369, in forward xyz, state, alpha = self.str2str(msa.float(), pair.float(), xyz.detach().float(), state.float(), idx, top_k=0) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast return func(args, kwargs) File "/home/rosetta/RoseTTAFold2NA/network/Track_module.py", line 299, in forward shift = self.se3(G, node.reshape(BL, -1, 1), l1_feats, edge_feats) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/rosetta/RoseTTAFold2NA/network/SE3_network.py", line 89, in forward return self.se3(G, node_features, edge_features) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/se3_transformer-1.0.0-py3.10.egg/se3_transformer/model/transformer.py", line 163, in forward File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/se3_transformer-1.0.0-py3.10.egg/se3_transformer/model/basis.py", line 166, in get_basis File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/contextlib.py", line 135, in enter return next(self.gen) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/cuda/nvtx.py", line 89, in range range_push(msg.format(args, **kwargs)) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/cuda/nvtx.py", line 31, in range_push return _nvtx.rangePushA(msg) File "/home/rosetta/miniconda3/envs/RF2NA/lib/python3.10/site-packages/torch/cuda/nvtx.py", line 10, in _fail raise RuntimeError( RuntimeError: NVTX functions not installed. Are you sure you have a CUDA build?

jessica-andreani commented 8 months ago

Hello, Not sure if it will work for you but I have experienced issues with NVTX and GPU too.

I have solved them by using a modified conda environment with CUDA 11.6 and cudatoolkit from nvidia instead of conda-forge

Also, I have to systematically run export CUDA_VISIBLE_DEVICES=0 once I have activated the conda environment and before running the RF2NA program.

Hope it helps.

fglaser commented 8 months ago

Dear Jessica

Thanks a lot for your kind answer,

Could you please detail a little how exactly did you proceed?

Regards, Fabian

Fabian Glaser, PhD

Structural and Computational Biology Unit The Lorry I. Lokey Center for Life Sciences and Engineering Technion - Israel Institute of Technology, Haifa, Israel

On 2 Dec 2023, at 15:58, Jessica Andreani @.***> wrote:

Hello, Not sure if it will work for you but I have experienced issues with NVTX and GPU too.

I have solved them by using a modified conda environment with CUDA 11.6 and cudatoolkit from nvidia instead of conda-forge

nvidia::cudatoolkit=11.6 dglteam::dgl-cuda11.6 Also, I have to systematically run export CUDA_VISIBLE_DEVICES=0 once I have activated the conda environment and before running the RF2NA program.

Hope it helps.

— Reply to this email directly, view it on GitHub https://github.com/uw-ipd/RoseTTAFold2NA/issues/36#issuecomment-1837154977, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSBVSC5CPVWIVNXC3QHHELYHMXX7AVCNFSM6AAAAAAS7HETVKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGE2TIOJXG4. You are receiving this because you commented.

jessica-andreani commented 7 months ago

Dear Fabian, sorry, I realized I do not have the latest version of the yml file. I had started from this earlier version https://github.com/uw-ipd/RoseTTAFold2NA/blob/43bdd8908236935df74528bfd3b787a493a5d251/RF2na-linux.yml and substituted two lines:

This solved my library errors, but I haven't tried with the new yml file yet.

Also, if I did not run the command indicated before (export CUDA_VISIBLE_DEVICES=0) in the terminal before running RF2NA, my GPU card was not detected.

Best regards Jessica

stianale commented 7 months ago

Jessica, maybe I misunderstand something here, but you say that you have to run the command you indicated, unless your GPU card was not detected. Isn't the whole point of this thread to get the program run without the need for GPU?

I find it a huge drawback that you still need a specific GPU card for RF2NA to run. Also, there are no real alternatives to running the program locally, no Colab notebook, nothing.

jessica-andreani commented 7 months ago

Hello, sorry if I wasn't clear. The program runs on CPU by default. We had trouble getting it to run on GPU, because even though we had an available GPU, it was not detecting it until I found that I need to specify this "export CUDA_VISIBLE_DEVICES=0". We observed huge speedup on GPU vs CPU (on one example standard size complex, a factor of 25 in runtime when using an NVIDIA A100 GPU vs. 20 CPU)

stianale commented 7 months ago

No problem.

So I should be able to run it on CPU while not having an NVIDIA GPU, nor cuda?

stianale commented 7 months ago

For those still struggling with the newest yml file, I think I stumbled upon a solution that may work universally.

Before doing anything, run this:

conda config --set channel_priority flexible

Based on https://pytorch.org/blog/deprecation-cuda-python-support/, I substituted pytorch for pytorch::pytorch=2.0 in the yml file, because just pytorch installs pytorch 2.1.*. So my yml file looks like this: 

name: RF2NA
channels:
  - pytorch
  - nvidia
  - defaults
  - conda-forge
dependencies:
  - python=3.10
  - pip
  - pytorch::pytorch=2.0
  - requests
  - pytorch-cuda=11.7
  - dglteam/label/cu117::dgl
  - pyg::pyg
  - bioconda::mafft
  - bioconda::hhsuite
  - bioconda::blast
  - bioconda::hmmer>=3.3
  - bioconda::infernal
  - bioconda::cd-hit
  - bioconda::csblast
  - pip:
    - psutil
    - tqdm

Then go about with conda env create -f RF2na-linux.yml

Finally, probably unnecessary to say, but after having run conda activate RF2NA:

cd SE3Transformer pip install --no-cache-dir -r requirements.txt python setup.py install

samhimes92 commented 6 months ago

@stianale that worked great thank you.

After doing your solution I got an error that said

ModuleNotFoundError: No module named 'pandas'

A simple pip install pandas solved that problem. It's working great now.

I wonder if pandas should be in the .yml file? Did anyone else have install pandas on their own?

boesr commented 5 months ago

@samhimes92

I get the same error with the version released on April, 14th. Going to install pandas as well and see if this fixed it

boesr commented 5 months ago

That fixed the pandas error. I also tried the approach of @stianale. It is still not run on the gpu. Still have issues running the example though. @samhimes92 did you change anything else?

/my/path/installations/021358/all/RoseTTAFold2NA/0.2/input_prep/make_rna_msa.sh: line 9: [: missing `]'
BLAST Database error: Error: Not a valid version 4 database.
rm: cannot remove 'blastn1.list.split.*': No such file or directory
BLAST Database error: Error: Not a valid version 4 database.
rm: cannot remove 'blastn2.list.split.*': No such file or directory

Error: Failed to autodetect format for target sequence database db
samhimes92 commented 5 months ago

@boesr I'm sorry to that it is giving you trouble. Unfortunately I didn't change anything else.

boesr commented 5 months ago

For everyone else having the same error. The RNA database was corrupt due to ASCI characters. I exchanged the installation command to gunzip -c rnacentral_species_specific_ids.fasta.gz | perl -ne 's/[^\x00-\x7F]+/ /g; print;' | makeblastdb -in - -dbtype nucl -parse_seqids -out rnacentral.fasta -title "RNACentral"

That does run now. It also uses the GPU (because of the other errors I never reached the "gpu part" of the test script)

SuhasSrinivasan commented 3 months ago

Hello everyone, please see https://github.com/uw-ipd/RoseTTAFold2NA/issues/105 for potential solution.

Tim15-tech commented 2 weeks ago

I also had this issue and want to add some points: The predict.py checks if torch.cuda.is_available evaluates to true. In case it is false, CPU is used and a message printed to the terminal. Therefore, within the RF2NA-env python import torch and testing for torch.cuda.is_available() already needs to evaluate to true. You can check the highest supported cuda version e.g. via nvidia-smi.

However, the new environment will only work if the RF2NA name is used and not a new name like CudaTestEnv. The bash-script must be run out of the base env and then activates the RF2NA env.

I added a (for me working) yml file in my fork including comments.

anar-rzayev commented 1 week ago

For everyone else having the same error. The RNA database was corrupt due to ASCI characters. I exchanged the installation command to gunzip -c rnacentral_species_specific_ids.fasta.gz | perl -ne 's/[^\x00-\x7F]+/ /g; print;' | makeblastdb -in - -dbtype nucl -parse_seqids -out rnacentral.fasta -title "RNACentral"

That does run now. It also uses the GPU (because of the other errors I never reached the "gpu part" of the test script)

First of all, thank you very much for this suggestion. After having Error: (803.7) [makeblastdb] Blast-def-line-set.E.title in the previous download, now this time it is gone but FASTA-Reader: Ignoring invalid residues at position(s) still occurs:

(RF2NA) [protein@node01 trial]$ gunzip -c rnacentral_species_specific_ids.fasta.gz | perl -ne 's/[^\x00-\x7F]+/ /g; print;' | makeblastdb -in - -dbtype nucl  -parse_seqids -out rnacentral.fasta -title "RNACentral"

Building a new DB, current time: 07/23/2024 11:11:55
New DB name:   /home/intern/protein/RoseTTAFold2NA/RNA/trial/rnacentral.fasta
New DB title:  RNACentral
Sequence type: Nucleotide
Keep MBits: T
Maximum file size: 1000000000B
FASTA-Reader: Ignoring invalid residues at position(s): On line 105003909: 10
FASTA-Reader: Ignoring invalid residues at position(s): On line 105004664: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105005148: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105005321: 6
FASTA-Reader: Ignoring invalid residues at position(s): On line 105005559: 8, 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 105005654: 17
FASTA-Reader: Ignoring invalid residues at position(s): On line 105006513: 24
FASTA-Reader: Ignoring invalid residues at position(s): On line 105006515: 24
FASTA-Reader: Ignoring invalid residues at position(s): On line 105007433: 21
FASTA-Reader: Ignoring invalid residues at position(s): On line 105007492: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105007789: 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 105007917: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105009135: 17
FASTA-Reader: Ignoring invalid residues at position(s): On line 105009138: 17
FASTA-Reader: Ignoring invalid residues at position(s): On line 105009195: 17
FASTA-Reader: Ignoring invalid residues at position(s): On line 105009229: 5-8
FASTA-Reader: Ignoring invalid residues at position(s): On line 105009786: 2-3, 6, 8, 11-13, 19, 24, 26-27, 33, 35-37, 41, 45-46, 49-52, 54, 58, 60
FASTA-Reader: Ignoring invalid residues at position(s): On line 105009787: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105010241: 4
FASTA-Reader: Ignoring invalid residues at position(s): On line 105010725: 4
FASTA-Reader: Ignoring invalid residues at position(s): On line 105010778: 23
FASTA-Reader: Ignoring invalid residues at position(s): On line 105010907: 9, 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 105011180: 7
FASTA-Reader: Ignoring invalid residues at position(s): On line 105011571: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105011651: 34
FASTA-Reader: Ignoring invalid residues at position(s): On line 105012150: 5
FASTA-Reader: Ignoring invalid residues at position(s): On line 105012152: 5
FASTA-Reader: Ignoring invalid residues at position(s): On line 105012647: 4
FASTA-Reader: Ignoring invalid residues at position(s): On line 105013026: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105013103: 8
FASTA-Reader: Ignoring invalid residues at position(s): On line 105013105: 8
FASTA-Reader: Ignoring invalid residues at position(s): On line 105013422: 27
FASTA-Reader: Ignoring invalid residues at position(s): On line 105013861: 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 105013870: 28
FASTA-Reader: Ignoring invalid residues at position(s): On line 105014518: 18
FASTA-Reader: Ignoring invalid residues at position(s): On line 105014744: 8
FASTA-Reader: Ignoring invalid residues at position(s): On line 105014792: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 105014875: 36
FASTA-Reader: Ignoring invalid residues at position(s): On line 105218513: 1-28, 32-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 105218514: 1-8
FASTA-Reader: Ignoring invalid residues at position(s): On line 105218529: 3, 12, 20-21, 25-26, 31
FASTA-Reader: Ignoring invalid residues at position(s): On line 105218609: 1-28, 32-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 105218610: 1-4
FASTA-Reader: Ignoring invalid residues at position(s): On line 105218612: 2, 4, 7-9, 15, 20, 22-23, 29, 31-33, 37, 41-42, 45-48, 50
FASTA-Reader: Ignoring invalid residues at position(s): On line 109558438: 4
FASTA-Reader: Ignoring invalid residues at position(s): On line 109558439: 4
FASTA-Reader: Ignoring invalid residues at position(s): On line 109558802: 6
FASTA-Reader: Ignoring invalid residues at position(s): On line 122325170: 34-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 122325171: 1-25
FASTA-Reader: Ignoring invalid residues at position(s): On line 122325335: 7
FASTA-Reader: Ignoring invalid residues at position(s): On line 122325458: 34-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 122325459: 1-24
FASTA-Reader: Ignoring invalid residues at position(s): On line 133425107: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 133425159: 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 133425597: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 133425638: 8
FASTA-Reader: Ignoring invalid residues at position(s): On line 133425759: 12
FASTA-Reader: Ignoring invalid residues at position(s): On line 133425827: 18
FASTA-Reader: Ignoring invalid residues at position(s): On line 139352472: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 139352515: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 147156248: 8
FASTA-Reader: Ignoring invalid residues at position(s): On line 147156605: 10
FASTA-Reader: Ignoring invalid residues at position(s): On line 147156952: 10-19
FASTA-Reader: Ignoring invalid residues at position(s): On line 153885193: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 153885437: 5
FASTA-Reader: Ignoring invalid residues at position(s): On line 153885464: 39, 43
FASTA-Reader: Ignoring invalid residues at position(s): On line 153885524: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 161492210: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 163976520: 41, 45
FASTA-Reader: Ignoring invalid residues at position(s): On line 163976526: 5
FASTA-Reader: Ignoring invalid residues at position(s): On line 173910357: 16
FASTA-Reader: Ignoring invalid residues at position(s): On line 173910360: 16
FASTA-Reader: Ignoring invalid residues at position(s): On line 173910539: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 173910805: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 173910898: 16
FASTA-Reader: Ignoring invalid residues at position(s): On line 173911120: 5
FASTA-Reader: Ignoring invalid residues at position(s): On line 173911213: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 173911236: 50
FASTA-Reader: Ignoring invalid residues at position(s): On line 173911269: 16
FASTA-Reader: Ignoring invalid residues at position(s): On line 173911562: 39
FASTA-Reader: Ignoring invalid residues at position(s): On line 184513076: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 184513250: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 184513300: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 184513304: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 184513391: 39
FASTA-Reader: Ignoring invalid residues at position(s): On line 185642062: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 238415857: 10
FASTA-Reader: Ignoring invalid residues at position(s): On line 238416024: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 247878720: 44
FASTA-Reader: Ignoring invalid residues at position(s): On line 247878904: 35
FASTA-Reader: Ignoring invalid residues at position(s): On line 247878949: 10
FASTA-Reader: Ignoring invalid residues at position(s): On line 248524638: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 248524747: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 248524824: 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 248525359: 17
FASTA-Reader: Ignoring invalid residues at position(s): On line 336283612: 2-4, 8, 11-13, 16-17, 20, 25, 32-33, 39-43, 45, 47-51, 54-56, 59-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 336283613: 1-2, 6-8, 12, 14-15
FASTA-Reader: Ignoring invalid residues at position(s): On line 336284583: 35, 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 336284630: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 336284639: 54
FASTA-Reader: Ignoring invalid residues at position(s): On line 336284647: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 336284916: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285098: 12
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285136: 1-2, 7, 12-13, 16, 18, 21, 25-26, 28-29, 33-35, 37, 39-42, 48, 50-51, 54-55, 57, 59-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285137: 1-6, 8, 10-11, 13-14
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285139: 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285141: 1-2, 7, 12-13, 16, 19, 23-24, 26-27, 31-33, 35, 37-40, 46, 48-49, 52-53, 55, 57-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285142: 1-4, 6, 8-9, 11-12
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285170: 42
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285341: 14-17
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285468: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285470: 18
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285804: 1-2, 7, 12-13, 16-17, 21, 25-26, 28-29, 33-35, 37, 39-42, 48, 50-51, 54-55, 57, 59-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285805: 1-6, 8, 10-11, 13-14
FASTA-Reader: Ignoring invalid residues at position(s): On line 336285809: 37
FASTA-Reader: Ignoring invalid residues at position(s): On line 336286784: 25
FASTA-Reader: Ignoring invalid residues at position(s): On line 336287181: 55
FASTA-Reader: Ignoring invalid residues at position(s): On line 336287319: 42
FASTA-Reader: Ignoring invalid residues at position(s): On line 336287340: 2-4, 8, 11-13, 16-17, 20, 25, 32-33, 39-43, 45, 47-51, 54-56, 59-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 336287341: 1-2, 6-8, 12, 14-15
FASTA-Reader: Ignoring invalid residues at position(s): On line 350331181: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 350331249: 2-4, 8, 11-13, 16-17, 20, 25, 32-33, 39-43, 45, 47-51, 54-56, 59-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 350331250: 1-2, 6-8, 12, 15
FASTA-Reader: Ignoring invalid residues at position(s): On line 350331318: 10
FASTA-Reader: Ignoring invalid residues at position(s): On line 350332168: 11
FASTA-Reader: Ignoring invalid residues at position(s): On line 350332337: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 350332363: 49
FASTA-Reader: Ignoring invalid residues at position(s): On line 350333269: 9
FASTA-Reader: Ignoring invalid residues at position(s): On line 350333416: 11
FASTA-Reader: Ignoring invalid residues at position(s): On line 350333637: 49
FASTA-Reader: Ignoring invalid residues at position(s): On line 351023421: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 351023427: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 351023429: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 351023446: 33
FASTA-Reader: Ignoring invalid residues at position(s): On line 351023528: 34
FASTA-Reader: Ignoring invalid residues at position(s): On line 351023942: 43
FASTA-Reader: Ignoring invalid residues at position(s): On line 351024104: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 351024223: 9
FASTA-Reader: Ignoring invalid residues at position(s): On line 351024821: 3-4
FASTA-Reader: Ignoring invalid residues at position(s): On line 351024907: 53
FASTA-Reader: Ignoring invalid residues at position(s): On line 351024990: 38
FASTA-Reader: Ignoring invalid residues at position(s): On line 353623246: 3
FASTA-Reader: Ignoring invalid residues at position(s): On line 353623255: 54
FASTA-Reader: Ignoring invalid residues at position(s): On line 353627905: 14-15
FASTA-Reader: Ignoring invalid residues at position(s): On line 474451276: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 474464277: 51
FASTA-Reader: Ignoring invalid residues at position(s): On line 474464308: 21
FASTA-Reader: Ignoring invalid residues at position(s): On line 474464657: 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 474464746: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 474464864: 1-46
Ignoring sequence 'lcl|URS0002617765_32630' as it has no sequence data
FASTA-Reader: Ignoring invalid residues at position(s): On line 474464897: 1-60
FASTA-Reader: Ignoring invalid residues at position(s): On line 474464898: 1-20
Ignoring sequence 'lcl|URS000261776D_32630' as it has no sequence data
FASTA-Reader: Ignoring invalid residues at position(s): On line 481118227: 5
FASTA-Reader: Ignoring invalid residues at position(s): On line 481118231: 2
FASTA-Reader: Ignoring invalid residues at position(s): On line 481118246: 10
FASTA-Reader: Ignoring invalid residues at position(s): On line 481118259: 5
FASTA-Reader: Ignoring invalid residues at position(s): On line 481118413: 34
FASTA-Reader: Ignoring invalid residues at position(s): On line 481118815: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 481118926: 25
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119309: 47
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119385: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119387: 6
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119507: 53
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119548: 9
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119549: 27
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119600: 24
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119706: 38
FASTA-Reader: Ignoring invalid residues at position(s): On line 481119751: 4-6
FASTA-Reader: Ignoring invalid residues at position(s): On line 481120058: 13
FASTA-Reader: Ignoring invalid residues at position(s): On line 481120093: 3
FASTA-Reader: Ignoring invalid residues at position(s): On line 481120102: 54
FASTA-Reader: Ignoring invalid residues at position(s): On line 481121154: 38
FASTA-Reader: Ignoring invalid residues at position(s): On line 481121184: 50
FASTA-Reader: Ignoring invalid residues at position(s): On line 481121246: 34
FASTA-Reader: Ignoring invalid residues at position(s): On line 481121563: 51
FASTA-Reader: Ignoring invalid residues at position(s): On line 481121598: 51
FASTA-Reader: Ignoring invalid residues at position(s): On line 498876809: 43
FASTA-Reader: Ignoring invalid residues at position(s): On line 498876844: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 515370907: 39
FASTA-Reader: Ignoring invalid residues at position(s): On line 515370946: 48
FASTA-Reader: Ignoring invalid residues at position(s): On line 515370983: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371048: 38
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371133: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371160: 43
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371392: 4
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371654: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371708: 55
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371801: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371803: 34
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371863: 28
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371877: 34, 39
FASTA-Reader: Ignoring invalid residues at position(s): On line 515371971: 1
FASTA-Reader: Ignoring invalid residues at position(s): On line 515372088: 37
Adding sequences from FASTA; added 37944865 sequences in 1862.13 seconds.

Do you know how to resolve those invalid residues as well?