CUDA OOM

[majen-dev]

Hi, I have this code:

import os
from pathlib import Path

from imagededup.methods import CNN

cnn_encoder = CNN()
image_dir = Path("**************************************************")
for item in os.listdir(image_dir):
    current_path = str(image_dir) + "/" + item
    if os.path.isdir(current_path):
        encode = cnn_encoder.encode_images(image_dir=Path(current_path), recursive=False, num_enc_workers=0)
        duplicates_cnn = cnn_encoder.find_duplicates_to_remove(encoding_map=encode, num_sim_workers=0, min_similarity_threshold=0.85, outfile="results.json")

        for duplicate in duplicates_cnn:
            os.remove(duplicate)

It tries to run on my 12GB VRAM GPU (RTX 3060) but OOMs basically immediately. The images are 1080p if that matters, and only 700 of them is being processed at once. Stack trace:

/home/pc/PycharmProjects/dan2023/.venv/bin/python /home/pc/PycharmProjects/dan2023/dedup.py /home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. warnings.warn( /home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=MobileNet_V3_Small_Weights.IMAGENET1K_V1. You can also use weights=MobileNet_V3_Small_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) 2024-01-23 15:01:24,823: INFO Initialized: MobileNet v3 pretrained on ImageNet dataset sliced at GAP layer 2024-01-23 15:01:24,842: INFO Device set to cuda .. 2024-01-23 15:01:24,988: INFO Start: Image encoding generation Traceback (most recent call last): File "/home/pc/PycharmProjects/dan2023/dedup.py", line 11, in encode = cnn_encoder.encode_images(image_dir=Path(current_path), recursive=False, num_enc_workers=0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/imagededup/methods/cnn.py", line 251, in encode_images return self._get_cnn_features_batch(image_dir=image_dir, recursive=recursive, num_workers=num_enc_workers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/imagededup/methods/cnn.py", line 146, in _get_cnn_features_batch arr = self.model(ims.to(self.device)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/imagededup/utils/data_generator.py", line 77, in forward return self.mobilenet_gap_op(x) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) ^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) ^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) ^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/modules/batchnorm.py", line 171, in forward return F.batch_norm( ^^^^^^^^^^^^^ File "/home/pc/PycharmProjects/dan2023/.venv/lib/python3.11/site-packages/torch/nn/functional.py", line 2478, in batch_norm return torch.batch_norm( ^^^^^^^^^^^^^^^^^ torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 50.00 MiB. GPU 0 has a total capacty of 11.76 GiB of which 70.12 MiB is free. Including non-PyTorch memory, this process has 10.06 GiB memory in use. Of the allocated memory 9.84 GiB is allocated by PyTorch, and 92.39 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[Blossomers]

Same. May I know if you have solved it?

[Snow-Pigeon]

I encountered the same issue. I found that the version I installed using pip install imagededup was 0.3.1, while the latest version on GitHub is 0.3.2. So, I deleted version 0.3.1, downloaded the source code from GitHub, and installed it via python setup.py install. After that, the issue was resolved.

I hope this helps you, @JilekJosef @Blossomers.

[sezan92]

is this problem solved?