Closed FX-STAR closed 3 years ago
Hi @whoNamedCody, would using smaller batch_size helps? I guess it might be caused by out of CUDA memory?
I'm sure Memory-Usage not full, and training on single gpu is normal
The stuck sometimes is caused by the different computation graphs in different GPUS and this usually happens when you are training empty GT images. You may check that first.
The stuck sometimes is caused by the different computation graphs in different GPUS and this usually happens when you are training empty GT images. You may check that first.
Hi, @ZwwWayne, I meet the same problem.
when I set filter_empty_gt=False (because there are many empty GT images in my custom dataset), dist_train for FCOS also keep waiting, so is there any way and plan to solve it?
Thanks.
The stuck sometimes is caused by the different computation graphs in different GPUS and this usually happens when you are training empty GT images. You may check that first.
I meet the same problem in training resnest.Could you help us solve the problem
@ZwwWayne
The implementation might not be so robust, we will check it out.
My env: cuda10.2 torch==1.6.0 mmdetection==2.8.0 mmcv==1.2.4 After some iters the GPU-Util 100% but the process is always waiting Any advice? Thanks