search

airsplay / R2R-EnvDrop

PyTorch Code of NAACL 2019 paper "Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout"
MIT License
123 stars 25 forks source link

Hardware requirements? #6

Closed daqingliu closed 5 years ago

daqingliu commented 5 years ago

Hi, thanks for your awesome paper and codes. Recently I'm preparing to follow your work and I'm wondering what hardware it requires and how long it will cost while training? Thanks!

airsplay commented 5 years ago

Emmmm, I have not tested the minimal requirement of computational resource but I believe that GTX 1080 would be enough.

daqingliu commented 5 years ago

How about the training time?

airsplay commented 5 years ago

Speaker and Agent finishes in around 6 hours. Agent + Speaker + EnvDrop converges slowly, which takes around 1 day to converge.

daqingliu commented 5 years ago

Thank you very much :)

ZhuFengdaaa commented 5 years ago

I train Speaker and Agent on two RTX 2080Ti in parallel and it takes 1 day for 80000 iterations. The GPU usage is about 30~40 since sampling data in RL takes time. Half of my CPUs are idle and half of memory are free during training.

I come to ask if you know how to promote training efficiency and what is your suggestion?

airsplay commented 5 years ago

Yep, I am aware that the GPUs are not fully utilized. But I am sorry that I do not know the answer.

Previously, the speed bottleneck is CPU-GPU data transfer (it's quite slow on pageable memory) and I optimize it a lot. So I am not sure whether CPU-based sampling would speed it up.