Open jaymefosa opened 3 years ago
Looks about right 👀 from my experience
oh hm.. the official implementation (from the paper) is about 10,000 iterations / hr (10x faster) on a single GPU
Same with me. I am training on RTX 3090, image size 512 batch size 32 but it has been 5 days and training is still not finished for 100k iterations.
I'm on a GTX 1660 Ti, getting 4.20s/it approx..
Example: 650 training steps after 45:28 mins
Running 0.20.5 on pytorch 1.10.1 for CUDA 11.3 on Linux with an RTX 3090 gives me about 1.27it/s (or 0.8s/it) when training 512 image size on a source batch of 2500 images, which comes out to around 32 hours estimate for the default 150000 iterations.
OP's numbers look OK to me for the hardware. I imagine the 'single GPU' figures in the README are probably for industrial cards like Quadros or so.
One thing to note is that this implementation has to load and process the images frequently during training. If your images are heavily compressed, way too large compared to your training image size, using a strange color space (like a palette or HDR mode), or are just on a slow disk, it might be impacting the performance.
In my case, all of my images are preprocessed to be 512x512 RGB PNGs in a low compression mode, and they're stored on an M.2 SSD, so they should be quick to read and decode for training batches. If your numbers are unusually slow, try seeing if there's an issue with your training data that's leaving your GPU waiting.
CPU: i7-9750H GPU: 2070
Default run settings. img-size 256. Checked it on another system with a 2080ti and had similarly long times. Is there an issue somewhere?