There seems to be a memory leak while encoding with CNN.
I encoded 10k images using CNN. The size of the images was 1000x1000. The memory usage was about 120GB. The pickled output of encodings was about 24MB. Filenames are 15 characters.
This is the code:
from imagededup.methods import CNN
hasher = CNN()
encodings = hasher.encode_images(image_dir='/images') # 24MB pickled, 120GB in memory.
There seems to be a memory leak while encoding with CNN.
I encoded
10k images
usingCNN
. The size of the images was1000x1000
. The memory usage was about120GB
. The pickled output of encodings was about 24MB. Filenames are 15 characters.This is the code: