search

yikang-li / FactorizableNet

Factorizable Net (Multi-GPU version): An Efficient Subgraph-based Framework for Scene Graph Generation
216 stars 38 forks source link

multi-gpu #18

Closed dorarad closed 5 years ago

dorarad commented 5 years ago

Hi,

Few questions regarding the multi-gpu settings:

  1. What should be the command to run the model with multi-gpu?
  2. Have you tested the model gets the same performance in the multi-gpu settings? (Asking in particular because the readme stresses that this is beta version so not sure, also I believe that recent commit also was about updating pytorch version and saw couple of other updates)
  3. In the standard single-gpu setting, how much time did it take you to train 1 epoch? I noticed that the gpu utilization is very low and was wondering maybe it's because of the recent multi-gpu commit?

In particular, I get a couple of warnings when trying to run it with the command on the readme page:

/juice/u/nlp/packages/anaconda_slurm/envs/dorarad-py2/lib/python2.7/site-packages/torch/nn/_reduction.py:47: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
/juice/u/nlp/packages/anaconda_slurm/envs/dorarad-py2/lib/python2.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '

Thanks a lot!!

yikang-li commented 5 years ago

Hi,

  1. add CUDA_VISIBLE_DEVICES=0,1 to specify the GPU you want to use. It will automatically use the specified devices and set the batch_size to the number of devices you use.
  2. I roughly tested the code. It can output comparable results. However, in our previous testing, the accelerate rate is not as expected (it looks like the multiple GPUs do not bring many improvements). We are still investigating that part.
  3. I think it takes 1~2 hours. Maybe I recommend using small dataset during explorative experiments.

Hope these help!

dorarad commented 5 years ago

Alright Thanks a lot for the info!