Giodiro
commented
3 years ago Hi @theophilec ! Thanks for the bug report!
In trying to reproduce the OOM I set up an environment similar to yours (main difference is GPU model and CUDA 11.1 instead of 11.0), but cannot seem to trigger the out-of-memory. This being said, I remember running into issues like this (memory not freed between runs) before, so I am quite confident that with a bit more help from your side the issue should become somewhat clearer.
If possible could you try to run the same code with a few more debugging statements added in. Something like this would be helpful:
import torch
from falkon.kernels import LinearKernel
from falkon import Falkon, FalkonOptions
n = 50000
d = 51000
l = 10
X = torch.randn((n, d))
y = torch.nn.functional.one_hot(torch.randint(0, l, (n,))).float()
opt = FalkonOptions(debug=True)
sigma = 1
penalties = [1e-4, 1e-5]
for i in range(2):
print(f"Fitting {i}")
kernel = LinearKernel(sigma=sigma, opt=opt)
model = Falkon(
kernel=kernel,
penalty=penalties[i],
M=40000,
maxiter=10,
seed=0,
options=opt,
)
model.fit(X, y)
predictions = model.predict(X)
print(f"Memory used after fit {i}")
for device in range(torch.cuda.device_count()):
print(torch.cuda.memory_summary(device))
# torch.cuda.empty_cache()
# If the line above is commented, FALKON induces a CUDA Out of Memory error.
theophilec
Hi FALKON team!
While using Falkon, I stumbled on what looks like a memory bug in the library.
Code to reproduce
Expected behavior
Fit different models, no errors.
Actual behavior
In the above code,
torch.cuda.empty_cache()line eliminates the issue.d = 51000tod = 30000eliminates the issue.Environment
pytorch 1.9pipLet me know if I can provide any further information or assistance in fixing the issue! Thanks!