When I was training, both train and val were normal, but when val saw a surge in video memory, and when I used four A100 video cards for training, I could only train one epoch, and then I was told that the video memory was insufficient. I don't know whether this code made video memory superposition indefinitely instead of releasing, or what was the problem? I don't know if you have encountered this kind of problem can tell me
ybk66
commented
1 month ago
When I was training, both train and val were normal, but when val saw a surge in video memory, and when I used four A100 video cards for training, I could only train one epoch, and then I was told that the video memory was insufficient. I don't know whether this code made video memory superposition indefinitely instead of releasing, or what was the problem? I don't know if you have encountered this kind of problem can tell me