Gpu memory usage keep increasing

[longzaitianguo]

When I trained the model on my own dataset with 300,000 images, the gpu memory usage kept increasing until it is used up which killed the training process. I am new to torch. Need your help to figure out this problem @da03

[da03]

Maybe try a smaller batch size. How much memory does your gpu have?

[longzaitianguo]

@da03 I used 32 for the batch size. The memory space is 12200MB. I solved this problem by resizing the image to the same height. Maybe the original images are too large. By the way, I found your data_gen code didn't resize the image while keeping the original image size. When I used your original data_gen code, the BLUE score is very low (like 0.1). After I changed the data_gen code to resize the image to the same height (like what you did in Attetion-ocr), the BLUE score increase to 0.95. What maybe the reason behind this? Thanks

[da03]

Are you using the raw images? That could be too large, we are cropping the images such that only the equations are kept (script preprocess_images.py).

[longzaitianguo]

@da03 Actually we already cropped the images by ourselves, so we did not use the preprocess_images.py. I wonder why you resize the image to the same height in the 'attention ocr' project while keeping the original size in this project?

[da03]

Hmm did you subsample the images to make the size X0.5 smaller? We keep the original size because this math dataset contains fractions, integrals and matrices, which has varying heights, while the attention ocr project mainly works on a single word, which has approximately the same height.