search

jryoungw / unet_3plus

Keras Implementation of U-Net 3+(UNet+++)
5 stars 5 forks source link

OOM in training 训练时发生OOM #1

Open yh0608 opened 3 years ago

yh0608 commented 3 years ago

您好，我在用这个网络训练时发生OOM，请问您的环境和设备是什么，。 我用的keras2.0.2，显卡是1080Ti 12G显存。 显存占用情况是10773M/11169M.

jryoungw commented 3 years ago

Can you write in English? I'm not a chinese speaker.

yh0608 commented 3 years ago

Can you write in English? I'm not a chinese speaker.

When I use this network to train , there was a Out Of Memory error occurred. keras version 2.0.2，input=(256,256,3), 1080Ti 12GB Memory. memory used in nvidia-smi is 10773M/11169M。 Thanks!

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[10,256,256,160] [[Node: conv2d_35/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](activation_4/Relu, conv2d_35/kernel/read)]]

jryoungw commented 3 years ago

I think you need larger GPU. 1080 Ti 12GB seems small for U-Net+++ to be trained.

yh0608 commented 3 years ago

I think you need larger GPU. 1080 Ti 12GB seems small for U-Net+++ to be trained.

Your suggestion is right. I changed batch_size from 10 to 5, it works! Thanks for your reply.