Open Co1lin opened 1 year ago
Probably a duplicate of #35600.
@sanchitintel Thanks for finding the similar issue! I checked my experiments again, and found that just calling jit trace for a same model could trigger this issue. I've corrected the statement above.
I think https://github.com/pytorch/pytorch/issues/35600 is a simpler case of mine because it just compiles the same model each time. It's worth to note that after the simpler one being fixed, this issue with compiling different models each time also needs to be fixed, because it's rare to compile a single model for many times, but it's common to compile many different models when doing neural architecture search. BTW, compiling different models means the models compiled before are "dead" (they are released by Python) but something related to them are not deleted.
Thanks for following up, @Co1lin!
BTW, compiling different models means the models compiled before are "dead"
If you mean there's a loop in which a variable is updated in each iteration, as in the example in #35600's description, then I think you're right!
(they are released by Python) but something related to them are not deleted
I believe you're right that references of objects of traced models are not being deleted somehow. I think fixing this issue would require diving deeper - perhaps Python Garbage collector API can be used to get refcounts, analyze the root-cause, and then subsequently fix the issue in PyTorch code. @voznesenskym might have more insights into this issue.
Just curious, are there any workaround/dirty fixes to route this issue while compiling multiple models in one process if it is hard to be fixed immediately? Thanks!
Found torch._C._jit_clear_class_registry
can somehow alleviate this issue but there is still some leakage.
Is there a potential fix in the works?
π Describe the bug
Hi! I find that running JIT trace for many different models in one process could lead to OOM. I think there are some memory leak problems here. I guess maybe currently torch doesn't delete a compiled graph even though its life cycle has ended.
I think this issue is worth fixing because in NAS (neural architecture search) people may do this.
Code to reproduce this issue:
Logs:
If we use the code below, the issue will disappear.
Versions