search

rfeinman / pytorch-minimize

Newton and Quasi-Newton optimization with PyTorch
https://pytorch-minimize.readthedocs.io
MIT License
308 stars 34 forks source link

Memory leak when optimizing a pytorch module using hess or hessp. #29

Open tatsuhiko-inoue opened 1 year ago

tatsuhiko-inoue commented 1 year ago

Hello,

When I ran "examples/train_mnist_Minimizer.py", the following warning was output.

/home/user/.pyenv/versions/py38-pytorch/lib/python3.8/site-packages/torch/autograd/__init__.py:200: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at ../torch/csrc/autograd/engine.cpp:1151.)  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass

I ran a script to optimize iteratively a pytorch module, and torch.cuda.OutOfMemoryError is occured.

pytorch version which I used is 2.0.1 and I used CUDA.

byfron commented 11 months ago

Found the same issue running a "dogleg" Minimizer for a non-linear least squares problem. Pytorch 1.13.1 with CUDA 11.6