fsx950223
commented
3 years ago NcclAllReduce only supports GPU.
Closed staeff777 closed 3 years ago
fsx950223
commented
3 years ago NcclAllReduce only supports GPU.
staeff777
commented
3 years ago Yes. This is due to the missing GPU information in the Subprocess. But this does not answer the question.
fsx950223
commented
3 years ago You should specific another GPU on subprocess or set gpu memory growth to True
fsx950223
commented
3 years ago Unimport TensorFlow in main process also works. Set CUDA_VISIBLE_DEVICES= in main process.
staeff777
commented
3 years ago I tried both.
config_proto.gpu_options.allow_growth = True. The problem remained the same. (I assume that this was the right way to to it)if __name__ == '__main__':
os.environ["CUDA_VISIBLE_DEVICES"]="-1"
import tensorflow.compat.v1 as tfThis only allows CPU detection, however the subprocess then also does not find GPUs, even if I set
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3" for the subprocess. It seems, that in this way around, the session is initialized for CPUs and does no longer "look" for GPU devices.
fsx950223
commented
3 years ago staeff777
commented
3 years ago Thank you. I thought about this too. Wouldn't this be - in this case- an equivalent to restarting the training script each epoch, since TF has to be initialized in each subprocess?
Thatswhy I was asking if I'm really the only one facing this problem. At the moment I'm training with commit #66995a6 - Handle the case of grad is None. ` where the memory leak did not yet appear.
I'll also try your Keras Model again. Its probably time to change. ;-)
fsx950223
commented
3 years ago import tensorflow
import multiprocessing
tf = tensorflow.compat.v1
config = tf.ConfigProto()
# Test with multiple threads invoking the one-shot iterator concurrently.
with tf.Graph().as_default() as g:
results = []
dataset = tf.data.Dataset.from_tensors([1, 2, 3]).map(lambda x: x * x)
iterator = tf.data.Dataset.make_one_shot_iterator(dataset)
next_element = iterator.get_next()
def consumer_thread():
with tf.Session(config=config, graph=g) as sess:
print(tf.test.is_gpu_available())
try:
print(sess.run(next_element))
except tf.errors.OutOfRangeError:
print('error')
p = multiprocessing.Process(target=consumer_thread, args=())
p.daemon=False
p.start()
p.join()
p2 = multiprocessing.Process(target=consumer_thread, args=())
p2.daemon=False
p2.start()
p2.join()Maybe this one is better.
staeff777
commented
3 years ago Thank you. I could run your example, but I could not adapt it to the TPUEstimator. I assume it's because the TPUEstimator creates its own session..
majnas
commented
3 years ago Hi, I have the same problem when using --strategy=gpus, anybody knows the solution!!?
fitoule
commented
2 years ago I have the same issue even with only one GPU :( But it works without Child Process option so my options are
It was working with TensorFlow until 2.5.2 but now efficientdet require TF > 2.8 so I am totally stuck
I'm trying a multi-gpu legacy model training using child processes to avoid the memory leak.
However, having the training in a seperate process causes following error:
The problem could be, that TF is already initiated in the parent process (according to an old source ). However, it seems that I'm the only one facing that problem, so maybe there is something that I'm doing wrong?