With each iteration, the model’s video memory gradually increases

[tongzhang111]

With each iteration, the model’s video memory gradually increases

[GX77]

I didn't quite understand your question, but I think it's normal for the memory usage to gradually increase in the early stages of training.

[tongzhang111]

With each iteration, the max mem gradually increases

[GX77]

I understand what you mean now, I didn't notice this situation before, I'm not quite sure about the reason, it might need careful debugging.

[GX77]

One speculation is that the image size is dynamic, which would lead to a gradual increase in video memory.

[tongzhang111]

As the number of iterations increases, the maximum memory also increases. Therefore, the problem of video memory explosion will occur. I determine whether there is a memory leak

[GX77]

If the OOM (Out of Memory) error occurs, you may need to reduce the resolution.