Open shalemeghana opened 6 months ago
Hello, thanks for the detailed bug report. The bug is likely due to GPU 1 out of memory.
I would suggest keep the memory
to the actual memory GPU1 has and set the capacity
arguments to its default value 42. This will probably solve the issue.
Also, the value for max_confs
was 1 in the original parameter file. Was there a specific reason for that? In this case, Auto3D will only generate 1 conformer for each molecule and optimize that one.
I am working on Auto3D on Linux based workstation to optimize and to generate the 3D structures. I have given a 500 compounds in .smi format. Here i have provided the parameter file which i have used in my case. parameter.txt I have used this command to run Auto3D engine.
After runing the above command it is showing this error.
RuntimeError: CUDA out of memory. Tried to allocate 4.27 GiB. GPU 1 has a total capacty of 15.74 GiB of which 11.88 MiB is free. Including non-PyTorch memory, this process has 15.71 GiB memory in use. Of the allocated memory 11.76 GiB is allocated by PyTorch, and 3.81 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF A clear and concise description of what the bug is.
And the complete output of the command is given below-
/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ ' """)
/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ ' """) /home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ ' """) Checking input file... /home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ ' """) There are 499 SMILES in the input file /home/sylab02/Auto3D_pkg/standard_smiles_2.smi. All SMILES and IDs are valid. Suggestions for choosing isomer_engine and optimizing_engine: Isomer engine options: RDKit and Omega. Optimizing engine options: AIMNET. The available memory is 60 GB. The task will be divided into 1 jobs. Job1, number of inputs: 499 /home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ ' """) /home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ ' """)
Isomer generation for job1 Enumerating cis/tran isomers for unspecified double bonds... Enumerating R/S isomers for unspecified atomic centers... Removing enantiomers... Stereo centers for 320 are not fully enumerated. Stereo centers for 382 are not fully enumerated. Enantiomers not removed for 54 Enumerating conformers/rotamers, removing duplicates... 100%|██████████████████████████████████| 13720/13720 [14:45:56<00:00, 3.87s/it]
Optimizing on job1 Preparing for parallel optimizing... (Max optimization steps: 10000) Total 3D conformers: 7502 0%| | 0/10000 [00:00<?, ?it/s] Process Process-5: Traceback (most recent call last): File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/auto3D.py", line 142, in optim_rank_wrapper optimizer.run() File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 426, in run optdict = ensemble_opt(ani, coord_padded, numbers_padded, charges, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 323, in ensemble_opt n_steps(state, param['opt_steps'], param['opttol'], param['patience']) File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 244, in n_steps e, f = state['nn'].forward_batched(coord, numbers, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 185, in forward_batched _e, _f = self(coord[batch], numbers[batch], charges[batch]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 146, in forward d = self.ani( ^^^^^^^^^ File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript, serialized code (most recent call last): File "code/torch/ensemble/___torch_mangle_4.py", line 27, in forward else: pass _out = (_0).forward(_in, )