Closed go-ahead-maker closed 1 year ago
By the way, If I launch use_checkpoint
, does it hurts the final performance?
use_checkpoint
will not hurt the final performance, but it decreases the training speed for saving GPU memory.
The leaked memory didn't exist in my experiments. Maybe there are some other bugs that need to be fixed.
Thanks for your quick reply.
I have checked the log file of swin. The memory
also increases during the training process (from the initial 7212 to 7909), and I also notice that it has slight changes in your log file (5162 to 5167). So I suppose that this phenomenon may be normal for mmdet.
In my experiments, I follow your config that the batch size is 4 4 (4 samples per GPU). Is this equivalent to 8 2?
Thanks again for your valuable reply!
Thanks for your try! If you keep the same total batch size, it should be equivalent.
As there is no more activity, I am closing the issue, don't hesitate to reopen it if necessary.
Hi, sorry for disturbing you again. I notice that the
use_checkpoint
is adopted in Uniformer for saving the GPU memory. Recently, I try to conduct some exp of my custom model for object detection, and I find the GPU consumption seems continually increase over time (every few iterations will add a few more), I'm not sure what's causing this. So, I wonder ifuse_checkpoint
could alleviate the problem. Does this(launchuse_checkpoint
) lead to slower training speed?Best.