search

moabitcoin / holy-edge

Holistically-Nested Edge Detection
https://arxiv.org/pdf/1504.06375.pdf
GNU General Public License v3.0
347 stars 137 forks source link

Memory #7

Open WangZhouTao opened 6 years ago

WangZhouTao commented 6 years ago

hello, I am a student.recently,I am reading the HED paper. I found that your code is very interesting,but some error is happened when I try to train the model with my own data.Error message show that Memory error.My computer in-build 32GB memory, would you give me some suggestion.Think you very much.

CangHaiQingYue commented 6 years ago

my GPU only has 6G memory which is fine to run the code. dont know your setting

WangZhouTao commented 6 years ago

Sorry,I forgot to tell you.I use 1080ti GPU and 32GB memory.I can run your code with my own data.But Memory error is happened when the iteration is about 13000.I obverse top command in ubuntu, i find Memory is insufficient（not gpu memory, is computer memory).If you could tell me some suggestion,I really appreciate what you’ve done。

CangHaiQingYue commented 6 years ago

Sorry, I have not had such error. This code will save a checkpoint each 100 steps, maybe 13000 is too big in your dataset( I trained at 30000 steps without memory error).

WangZhouTao commented 6 years ago

I already changed to save a checkpoint with each 1000 steps, but the error is still happen.Did you run this code on windows? Thank you.

CangHaiQingYue commented 6 years ago

I run it on Ubuntu 16.04.

sandhawalia commented 6 years ago

Could you please give details of your data-set ? It might be the case that your image sizes are quite large and since the VGG base model is full convolutional the intermediate representations overflow they GPU memory.

wutachiang commented 6 years ago

I also encountered the same problem with you, my server configuration is 8 Tesla p20, memory is 512G. Even with this configuration, memory error occurs after approximately 13,000 iterations during training. Will you solve the problem?@CangHaiQingYue

CangHaiQingYue commented 6 years ago

@Jasontachiangwu Well, when I cancelled the 'summary_write' operator, this problem was gone. So I guess there maybe some bug in 'summary'. You may rebuild your own code.

wutachiang commented 6 years ago

@CangHaiQingYue，After I updated the tensorflow version to r1.8, no problem was found after training. It may be that there is a bug in the summary.

yuezhilanyi commented 6 years ago

with same code on windows 10 and ubuntu 16.04, tensorflow v1.4.0, 20000 iterations: memory error occurs in ubuntu 16.04 while no error occurs in windows 10