search

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Apache License 2.0
15.5k stars 3.49k forks source link

*help* t2t-decoder uses up both CPU and GPU memory. #836

Open GuiyuDu opened 6 years ago

GuiyuDu commented 6 years ago

Description

When I use t2t-decoder to decode from dataset, this command takes up both CPU and GPU memory. I think that t2t-decoder did not use GPU. When I don't set worker_gpu_memory_fraction, CUDA_ERROE_OUT_OF_MEMORY occurs.

TensorFlow and tensor2tensor versions

tensor flow-gpu==1.5.0 & tensor2tensor==1.6.3

I tried to find out the reason from the source code but failed.Can anyone give me some help? Thanks.

martinpopel commented 6 years ago

I am not sure what is your problem. If you want to decode on CPU only, install tensorflow (i.e. not tensorflow-gpu) in a virtual env or hide the GPU with CUDA_VISIBLE_DEVICES. If you want to decode on GPU, you should expect it takes both CPU and GPU memory.

The default worker_gpu_memory_fraction should be OK. If you get OOM, you should decrease the batch_size (in decoding hparams, in case of decoding). BTW: I remember strange effects when adjusting worker_gpu_memory_fraction, something like that 0.91 and 0.93 was OK, but 0.92 failed (I don't remember the exact values).

GuiyuDu commented 6 years ago

I mean that t2t-decoder use CPU to decode rather than GPU. I follow the guide of the walkthrough and change nothing. When I use the default worker_gpu_memory_fraction, it told me that the allocation failed. But I use nvidia-smi to check the usage of memory, I found the memory has been allocated.

lkluo commented 6 years ago

You may kill the process on GPUs and retry.