search

wengong-jin / abdockgen

MIT License
104 stars 22 forks source link

RuntimeError: CUDA out of memory #1

Closed wjs20 closed 1 year ago

wjs20 commented 1 year ago

Hi

Thanks for uploading the code, this looks really interesting!

I have had some memory issues running your example, maybe you can help me?

it comes when I run the model.generate(epitope, surface) function

out = model.generate(epitope, surface)

RuntimeError: CUDA out of memory. Tried to allocate 120.00 MiB (GPU 0; 4.00 GiB total capacity; 3.09 GiB already allocated; 0 bytes free; 3.34 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I've put my system details below

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.81       Driver Version: 513.29       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA T500         On   | 00000000:01:00.0 Off |                  N/A |
| N/A   43C    P8    N/A /  N/A |   3848MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2175      C   /python3.9                      N/A      |
+-----------------------------------------------------------------------------+

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
alsa-lib                  1.2.7.2              h166bdaf_0    conda-forge
anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py39hb9d737c_2    conda-forge
asttokens                 2.0.8              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
biopython                 1.79             py39hb9d737c_2    conda-forge
blas                      2.116                       mkl    conda-forge
blas-devel                3.9.0            16_linux64_mkl    conda-forge
bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py39hb9d737c_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py39he91dace_1    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
contourpy                 1.0.5            py39hf939315_0    conda-forge
cryptography              38.0.2           py39hd97740a_0    conda-forge
cudatoolkit               11.7.0              hd8887f6_10    nvidia
cudnn                     8.4.1.50             hed8a83a_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.6.3            py39h5a03fae_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 1.1.1              pyhd8ed1ab_0    conda-forge
expat                     2.4.9                h27087fc_0    conda-forge
fftw                      3.3.10          nompi_hf0379b8_105    conda-forge
flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               hc2a2eb6_1    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.37.4           py39hb9d737c_0    conda-forge
freetype                  2.12.1               hca18f0e_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
glib                      2.74.0               h6239696_0    conda-forge
glib-tools                2.74.0               h6239696_0    conda-forge
gst-plugins-base          1.20.3               h57caac4_2    conda-forge
gstreamer                 1.20.3               hd4edc92_2    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4           py39hf3d152e_0    conda-forge
importlib_resources       5.10.0             pyhd8ed1ab_0    conda-forge
ipykernel                 6.16.1             pyh210e3f2_0    conda-forge
ipython                   8.5.0              pyh41d4057_1    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.0.2              pyhd8ed1ab_1    conda-forge
jack                      1.9.21               h2a1e645_0    conda-forge
jedi                      0.18.1             pyhd8ed1ab_2    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
jq                        1.6               h36c2ea0_1000    conda-forge
jsonschema                4.16.0             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0            py39hf3d152e_7    conda-forge
jupyter_client            7.4.3              pyhd8ed1ab_0    conda-forge
jupyter_console           6.4.4              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py39hf3d152e_0    conda-forge
jupyter_server            1.21.0             pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.3              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py39hf939315_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.39                 hc81fddc_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0            16_linux64_mkl    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcap                    2.66                 ha37c62d_0    conda-forge
libcblas                  3.9.0            16_linux64_mkl    conda-forge
libclang                  14.0.6          default_h2e3cab8_0    conda-forge
libclang13                14.0.6          default_h3a83d3e_0    conda-forge
libcups                   2.3.3                h3e49a29_2    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.1                h27087fc_0    conda-forge
libgcc-ng                 12.2.0              h65d4601_18    conda-forge
libgfortran-ng            12.2.0              h69a702a_18    conda-forge
libgfortran5              12.2.0              h337968e_18    conda-forge
libglib                   2.74.0               h7a41b64_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0            16_linux64_mkl    conda-forge
liblapacke                3.9.0            16_linux64_mkl    conda-forge
libllvm14                 14.0.6               he0ac6c6_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.38               h753d276_0    conda-forge
libpq                     14.5                 hd77ab85_0    conda-forge
libprotobuf               3.21.8               h6239696_0    conda-forge
libsndfile                1.1.0                h27087fc_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.39.4               h753d276_0    conda-forge
libstdcxx-ng              12.2.0              h46fd767_18    conda-forge
libtiff                   4.4.0                h55922b4_4    conda-forge
libtool                   2.4.6             h9c3ff4c_1008    conda-forge
libudev1                  249                  h166bdaf_4    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libuv                     1.44.2               h166bdaf_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvm-openmp               14.0.4               he0ac6c6_0    conda-forge
magma                     2.5.4                hc72dce7_3    conda-forge
markupsafe                2.1.1            py39hb9d737c_1    conda-forge
matplotlib                3.6.1            py39hf3d152e_0    conda-forge
matplotlib-base           3.6.1            py39hf9fd14e_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   2.0.4              pyhd8ed1ab_0    conda-forge
mkl                       2022.1.0           h84fe81f_915    conda-forge
mkl-devel                 2022.1.0           ha770c72_916    conda-forge
mkl-include               2022.1.0           h84fe81f_915    conda-forge
mpg123                    1.30.2               h27087fc_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.31               haf5c9bc_0    conda-forge
mysql-libs                8.0.31               h28c427c_0    conda-forge
nbclassic                 0.4.5              pyhd8ed1ab_0    conda-forge
nbclient                  0.7.0              pyhd8ed1ab_0    conda-forge
nbconvert                 7.2.2              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.2.2              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.2.2              pyhd8ed1ab_0    conda-forge
nbformat                  5.7.0              pyhd8ed1ab_0    conda-forge
nccl                      2.14.3.1             h0800d71_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
notebook                  6.5.1              pyha770c72_0    conda-forge
notebook-shim             0.2.0              pyhd8ed1ab_0    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.78                 h2350873_0    conda-forge
numpy                     1.23.4           py39h3d75532_0    conda-forge
oniguruma                 6.9.8                h166bdaf_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1q               h166bdaf_1    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandoc                    2.19.2               ha770c72_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.37                hc3806b6_1    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0            py39hd5dbb17_2    conda-forge
pip                       22.3               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
prometheus_client         0.15.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.31             pyha770c72_0    conda-forge
prompt_toolkit            3.0.31               hd8ed1ab_0    conda-forge
psutil                    5.9.3            py39hb9d737c_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio                14.0                habe0971_10    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7           py39h18e9c17_1    conda-forge
pyqt5-sip                 12.11.0          py39h5a03fae_1    conda-forge
pyrsistent                0.18.1           py39hb9d737c_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.9.13          h9a8a25e_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.12.1          cuda112py39hb0b7ed5_201    conda-forge
pyzmq                     24.0.1           py39headdf64_0    conda-forge
qt-main                   5.15.6               hc525480_0    conda-forge
qtconsole                 5.3.2              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.3.2              pyha770c72_0    conda-forge
qtpy                      2.2.1              pyhd8ed1ab_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                65.5.0             pyhd8ed1ab_0    conda-forge
sip                       6.7.2            py39h5a03fae_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.39.4               h4ff8645_0    conda-forge
stack_data                0.5.1              pyhd8ed1ab_0    conda-forge
tbb                       2021.6.0             h924138e_0    conda-forge
terminado                 0.16.0             pyh41d4057_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
torchvision               0.13.0          cuda112py39hd2c45b6_0    conda-forge
tornado                   6.2              py39hb9d737c_0    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.5.0              pyhd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022e                h191b570_0    conda-forge
unicodedata2              14.0.0           py39hb9d737c_1    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.4.1              pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.3              pyhd8ed1ab_0    conda-forge
xcb-util                  0.4.0                h166bdaf_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h166bdaf_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h166bdaf_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.9.0              pyhd8ed1ab_0    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge
wjs20 commented 1 year ago

It seemed to be an issue with my pytorch installation, I've tweaked it a bit and seems to be working now.

Closing issue.