Closed FabianSchuetze closed 1 year ago
Hello! Thanks for your issue to point out this problem, I was wondering, do you refine the max_iter
with a larger batch-size
To the best of my knowledge, detectron2 only support iter-based training now, so if you are using a larger training batch-size, you should set a small max_iter in your config, an example:
total_batch_size = 16
max_iter = 90000
total_batch_size = 32
max_iter = 45000
Thanks for the kind & informative reply!
Your answer makes a lot of sense - thanks.
Thanks for the kind & informative reply!
Your answer makes a lot of sense - thanks.
You're welcome~ I also believe that using epoch-based trainer may not be that confused~ we will try to update a epoch-based trainer in detrex in the future version
Hi,
thanks for the wonderful repo! It's a pleasure to read the code and to work with it.
The runtime of the training seems to increase dramatically when I increase the batch size, despite still having a good GPU utilization and the
data_time
just a small fraction of thetime
.For example, with a batch size of 4 (with maskdino) is see:
Whilst with a batch size of 2 I see:
My understanding would be that if
time
double but the batch size also double, the runtime should be identical. I am a bit surprised thattime
double despite thedata_time
being still so low (dataloader does not seem to be the bottleneck) and the GPU memory still be available and utilization is also not constantly maxed out. This is running on a single GPU with amp activated.Why could that be? Can I do something about it?