Closed staeff777 closed 3 years ago
NcclAllReduce only supports GPU.
Yes. This is due to the missing GPU information in the Subprocess. But this does not answer the question.
You should specific another GPU on subprocess or set gpu memory growth to True
Unimport TensorFlow in main process also works. Set CUDA_VISIBLE_DEVICES= in main process.
I tried both.
config_proto.gpu_options.allow_growth = True
. The problem remained the same. (I assume that this was the right way to to it)if __name__ == '__main__':
os.environ["CUDA_VISIBLE_DEVICES"]="-1"
import tensorflow.compat.v1 as tf
This only allows CPU detection, however the subprocess then also does not find GPUs, even if I set
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3"
for the subprocess. It seems, that in this way around, the session is initialized for CPUs and does no longer "look" for GPU devices.
Thank you. I thought about this too. Wouldn't this be - in this case- an equivalent to restarting the training script each epoch, since TF has to be initialized in each subprocess?
Thatswhy I was asking if I'm really the only one facing this problem. At the moment I'm training with commit #66995a6 - Handle the case of grad is None. ` where the memory leak did not yet appear.
I'll also try your Keras Model again. Its probably time to change. ;-)
import tensorflow
import multiprocessing
tf = tensorflow.compat.v1
config = tf.ConfigProto()
# Test with multiple threads invoking the one-shot iterator concurrently.
with tf.Graph().as_default() as g:
results = []
dataset = tf.data.Dataset.from_tensors([1, 2, 3]).map(lambda x: x * x)
iterator = tf.data.Dataset.make_one_shot_iterator(dataset)
next_element = iterator.get_next()
def consumer_thread():
with tf.Session(config=config, graph=g) as sess:
print(tf.test.is_gpu_available())
try:
print(sess.run(next_element))
except tf.errors.OutOfRangeError:
print('error')
p = multiprocessing.Process(target=consumer_thread, args=())
p.daemon=False
p.start()
p.join()
p2 = multiprocessing.Process(target=consumer_thread, args=())
p2.daemon=False
p2.start()
p2.join()
Maybe this one is better.
Thank you. I could run your example, but I could not adapt it to the TPUEstimator. I assume it's because the TPUEstimator creates its own session..
Hi, I have the same problem when using --strategy=gpus, anybody knows the solution!!?
I have the same issue even with only one GPU :( But it works without Child Process option so my options are
It was working with TensorFlow until 2.5.2 but now efficientdet require TF > 2.8 so I am totally stuck
I'm trying a multi-gpu legacy model training using child processes to avoid the memory leak.
However, having the training in a seperate process causes following error:
The problem could be, that TF is already initiated in the parent process (according to an old source ). However, it seems that I'm the only one facing that problem, so maybe there is something that I'm doing wrong?