About the training time

[liqikai9]

I run the following command with slurm:

python -m torch.distributed.launch --nproc_per_node=8 --master_port 29503 --use_env run/train_3d.py --cfg configs/panoptic/best_model_config.yaml

I run the code with 8 GPUs (Tesla V100 with 32GB memory) But it takes 5 days (5-04:29:24 exactly) to run 85 epochs. And I got the following results:

| Threshold/mm |   25  |   50  |   75  |  100  |  125  |  150  |
+--------------+-------+-------+-------+-------+-------+-------+
|      AP      | 88.04 | 96.05 | 96.89 | 97.38 | 97.59 | 97.76 |
|    Recall    | 91.17 | 96.84 | 97.52 | 97.87 | 98.07 | 98.22 |
| recall@500mm | 98.44 | 98.44 | 98.44 | 98.44 | 98.44 | 98.44 |
+--------------+-------+-------+-------+-------+-------+-------+
INFO:root:MPJPE: 16.80mm
INFO:root:=> saving checkpoint to /mnt/lustre/liqikai.vendor/open_mmlab/pose3d/mvp/output/panoptic/multi_person_posenet_50/best_model_config (Best: False)
Epoch: 85
current lr 1e-05

in which the MPJPE is 16.80mm, which is a little worse than the reported 15.8mm in your paper.

I have the following questions.

1. How long does it take to run your training code in your experiments? Could you please share the approximate training time as well as the GPU you used?

2. How many epochs does it take to converge to the best MPJPE or the evaluation metric you use during training? I see you need to train 200 epochs in the configs/panoptic/best_model_config.yaml. Is it necessary?

Thanks in advance and look forward to your reply!

[twangnh]

hi, please search the best result (Best: True) in your experiment log

[liqikai9]

How long did you take to get the best result? and how many epochs did it take to converge to the best result?

[twangnh]

it should be about 50 epochs, you can try longer time to get higher best result