search

GAMES-UChile / mogptk

Multi-Output Gaussian Process Toolkit
MIT License
164 stars 43 forks source link

Issue with model training #60

Closed sgalee2 closed 1 year ago

sgalee2 commented 1 year ago

I am re-running some experiments with a model I built in MOGPTK a few weeks back. Having changed nothing, I now encounter the error:

**RuntimeError**: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [200]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

This occurs when the backend calls torch.autograd.backward once my script calls model.train(Verbose=True).

Any help would be appreciated!

Thanks

tdewolff commented 1 year ago

Hi, thanks for reaching out! We found a similar issue lately in https://github.com/GAMES-UChile/mogptk/issues/53. What version of pytorch are you using? Do you have a small example that reproduces the error?

sgalee2 commented 1 year ago

I was running pytorch 2.0.0, I have since switched to 1.13.1 and everything works fine. I came up with the same error simply running a few of the example notebooks

tdewolff commented 1 year ago

Did you install through pip or from this repository? The bug fix hasn't been pushed to pip yet, but is in the repository.

EDIT: new released has been published, can you verify if it is fixed now?