Too much memory was cost for eval and train

[AnnaTrainingG]

🐛 Bug description

Problem: when I set persistent_worker = True in datalaoder, The memory cost of model will increase 2 times eg: the model is training 344G CPU memory was token, after one epoch, use _run_once_on_dataset for eval, 630G memory of CPU was costed, and the memory will never released after eval. so it'll be OOM when memory size of GPU is very small

Environment

• PyTorch Version :1.11
• ignite Version: 0.4.8
• OS : Linux
• How you installed Ignite (conda, pip, source):pip
• Python version: 3.8
• Any other relevant information:

[vfdev-5]

@niuliling123 thanks for reporting this issue! Do you think it could be possible to get a minimal repro code for that? At first glance, persistent_worker is related to pytorch dataloader and it may be possible that ignite has little impact on that. I would like to figure out whether it is pytorch or ignite problem.

[AnnaTrainingG]

I got it , thanks