ksopyla
commented
7 years ago Hi, I'm sorry that I reply so late, but I had github notification turn off. I work with dataset of 30k images in 304x64px resolution. All pics was convertet to gray scale, below is snippet which convert to gray and rescale to better resolution. I didn't have any memory issues while computing on CPU and GPU GeForce 780 (2GB RAM). If I understand correctly you have 150k images each 25x96px in RGB (3 channels): 15000025963/(10241024) = 1029.96MB - so it should fit in to your memory.
img = imread(path)
im_shape = img.shape
if len(im_shape)>2:
print('convert to gray shape={} file={}'.format(im_shape,file))
#convert to gray img
# r, g, b = img[:,:,0], img[:,:,1], img[:,:,2]
# gray = 0.2989 * r + 0.5870 * g + 0.1140 * b
#convert ot gray, faster
img = np.mean(img,-1)
else:
# each immage has size 57x300, we have to
# resize images to 64x304, because conv nets work better with
# dimensions divided by 2, we add 4 pixesl at the top
# 3 pixesl at the bottom and 2 to left and right
# TODO: change it to more automatic way, what if image size will be
im_pad = np.pad(img,((4,3),(2,2)), 'constant', constant_values=(255,))
Hi,
I'm doing some work about recognizing captcha by CNN. Your code and your answer helped me a lot.
Following your advice, I did normalization. So it needed load the whole dataset into memory.
My pc has 24G memory but run out of it. The model only trained 1 epoch and got stuck due to a lack of memory.
I wonder how much memory do you need in your work? Is there anything wrong in my code? My dataset is 150000*25*96*3 pics. How about the dataset you used?
Thanks!