How long does it take to complete a training?

[paolomandica]

Hey, just wanted to say thanks for the awesome paper and open-source code! It's been really helpful. I have a quick question about training times, and I'd be grateful if you could provide some insights.

Using batch_size=32 on 4 Tesla V100, each training cycle (120k iterations) takes approximately 10h + 5h for the active learning phase, which repeated for 5 times (active learning cycles) leads to a total of more than 3 days.

My questions are the following:

• Do these numbers seem correct to you? Should the training process take much less time?
• If I increase the BS and reduce the number of iterations, can I achieve the same results?

Thanks for your help! Paolo

[TheRevanchist]

Hi Paolo,

Yes, the timing sounds just right. I remember it was taking me around 3 days to do a full training (5 cycles).

Not sure about increasing the batch size (and lowering the number of iterations). It might require changing of the learning rate and the learning rate scheduler.

[paolomandica]

Got it! Thanks Ismail!