Closed tchaton closed 3 years ago
This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!
Hi @tchaton. I'm facing this memory leak. Returning an output in
validation_step
wil always be stored althoughvalidation_epoch_end
isn't defined. You can test it removingvalidation_epoch_end
from the BoringModel. For me, it's an unexpected behavior as I expect to behave validation loop as the training loop. That it's, if you don't want to store the outputs, do not overridetrain_epoch_end
hook.In my case, I had a callback to compute the metrics and log validation outputs to an external service. So, I return a dictionary in
validation_step
and implement callbacks that overrideson_validation_batch_end
andon_validation_epoch_end
. This causes that outputs are stored --> memory leak.Originally posted by @hal-314 in https://github.com/PyTorchLightning/pytorch-lightning/issues/8453#issuecomment-887508240