Open daisylab opened 6 years ago
@daisylab, did you ever figure out your issue? or has it still been slow?
@daisylab @tslater I have kind of similar issues, Are there any optimizations on the GPU that you can do to speed it up. I'm using the 4 Tesla V100 sxm2 16gb GPU's and the train step time & utilization is as below:
Step Time: 0.54s
Under Utilized:
Looping some folks here - @bastings @oahziur
Appreciate any help.Thanks.
I think there several reasons for this issue,
maybe you should set the batch_size
as large as possible to utilize your GPU
when you specify num_gpus
, nmt model will split the model, put different part in different GPU, this is not data parallelism, it's workload partitioning, you can make a data parallel training version instead of using num_gpus
flag
Hi, recently I began to study neural machine translation (seq2seq). Currently I'm reading a tutorial from TensorFlow.
https://www.tensorflow.org/tutorials/seq2seq
I ran the sample code from the site, i.e., the first NMT model, translating from Vietnamese to English. I have a machine with 2 nvidia 1080 Ti GPU. So I expected some speed-up from the two GPUs.
However, to my surprise, using two GPUs actually is slower than using one GPU.
Here's the command I used and the result.
And I could see step-time and wps as;
And when I used two GPUs;
I got only
So my question is:
I'm glad to hear about you.
Best regards, sungjin