a bug of "OOM, about gpu" when I run it on more than one spark worker node

[younfor]

for example : I had change the files to load my own pic like [None,32,32,3] . Everything is OK, but when I set the partition=2 or 4 , 8 ... and my computer information is gtx1070, ubuntu14.04, 8G. I also change the model init code with: `config = tf.ConfigProto() config.gpu_options.allow_growth = True config.gpu_options.allocator_type = 'BFC'

config.gpu_options.per_process_gpu_memory_fraction = 0.2

    session = tf.Session(config=config)`

upon will enable several process in one gpu. the bug is when the programer run some epoches , I find "nvidia-smi" 's gpu memory grows without stop. from 800MB to 2G , 4G, 8G... finally show some errors like cuda OOM. my way to solve it: after my check and try to fix it, I find a function leads to the GPU Memory Leak ` def reset_gradients(self):

with self.session.as_default():

     #self.gradients = [tf.zeros(g[1].get_shape()).eval() for g in self.compute_gradients]                                            
     self.gradients = [0.0]*len(self.compute_gradients)     # my modify                          
     self.num_gradients = 0

` though I don't the details why this change can works ,but it did.
email :younfor@yeah.net

[illuzen]

Interesting. I remember seeing this error sometimes, IIRC we installed newer drivers to fix it, but this seems like a pretty benign fix. I can kinda see why this might allocate more memory. In any case we shouldn't be running an eval just to get a bunch of zeros. Good catch, wanna roll this into the other PR?

Did you see a performance change when you set allow_growth = True? And do you have a cluster or is it a bunch of cores on a local?