Stonesjtu / pytorch_memlab

Profiling and inspecting memory in pytorch
MIT License
1.01k stars 37 forks source link

weakly-referenced object no longer exists #40

Open vsantjr opened 2 years ago

vsantjr commented 2 years ago

Hello,

Firstly, congratulations for memlab. I have been trying to use it in Google Colab, but sometimes this error happens:

ReferenceError Traceback (most recent call last)

in () 33 print('Reporter!!!!!!!') 34 reporter = MemReporter() ---> 35 reporter.report() 2 frames /usr/local/lib/python3.7/dist-packages/pytorch_memlab/mem_reporter.py in (.0) 62 #FIXME: make the grad tensor collected by gc 63 objects = gc.get_objects() ---> 64 tensors = [obj for obj in objects if isinstance(obj, torch.Tensor)] 65 for t in tensors: 66 self.device_mapping[t.device].append(t) ReferenceError: **weakly-referenced object no longer exists** In my code, I use MemReport() just right after the training phase, i.e.: `for epoch in range(num_epochs):` `net.train()` `...` `# end of training` print('Reporter!!!!!!!') reporter = MemReporter() reporter.report() Do you know what is the problem? Thank you and regards.
Stonesjtu commented 2 years ago

This error message is weird.

Could you plz post the:

btw could you try adding a torch.cuda.synchronize before creating a reporter.

vsantjr commented 2 years ago

Hello,

I am using Google Colab. I just ran !pip install pytorch_memlab in the notebook. The PyTorch version is the one of Colab. The GPU I guess that is K80 but I am not so sure.

Thank you.