Large crop size [1024, 2048] training with 32 gpus getting bad results

bowenc0221 / panoptic-deeplab

This is Pytorch re-implementation of our CVPR 2020 paper "Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation" (https://arxiv.org/abs/1911.10194)

Apache License 2.0

585 stars 117 forks source link

Large crop size [1024, 2048] training with 32 gpus getting bad results #68

Closed Jensen-Su closed 3 years ago

Jensen-Su commented 3 years ago

I have successfully reproduced the results with 512x1024 crop size. As following:

Config

Backbone: R52-DC5
Batchsize: 32
InitLr: 0.001
TrainedIters: 90k
CropSize: [512, 1024]
FrameWork: detectron2

Performance

PQ 60.7
mIou 78.8
AP 32.2

Then I simply changed the CropSize to [1024, 2048], and trained the model with 32 GPUs and batch size 32 for 90K iters. But the results were several points worse, like PQ lower than 57.
What did I miss?

bowenc0221 commented 3 years ago

Try 60k iterations when using crop size [1024, 2048]. I got something like this with larger crop size (R50 model).

mIoU	PQ	SQ	RQ	AP
79.6	61.9	81.5	74.9	30.4