search

GeorgeSeif / Semantic-Segmentation-Suite

Semantic Segmentation Suite in TensorFlow. Implement, train, and test new Semantic Segmentation models easily!
2.5k stars 880 forks source link

Error for Crop Height and Width of the FC-DenseNet103? #230

Open AI-ML-Enthusiast opened 4 years ago

AI-ML-Enthusiast commented 4 years ago

@GeorgeSeif Thanks for your awesome repository. I am training FCDenseNet-103 on my own dataset. It is fine with Crop_height and width=512, but when I change it to 1024, it gives the following error:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,288,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node FC-DenseNet103_3/denseblock3/BatchNorm_6/FusedBatchNorm (defined at D:\feasibilityAnalysis\RiversideProject\Code\Semantic-SegmentationTF\models\FC_DenseNet_Tiramisu.py:13) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

[[node Mean_3 (defined at D:/feasibilityAnalysis/RiversideProject/Code/Semantic-SegmentationTF/train.py:102) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

My environment: Windows 10 Tensorflow 1.13 python3.6 GPU

Is there any clue to solve this issue?

Thanks in Advanced