mpatacchiola
commented
2 years ago I just run the code on my machine multiple times and it worked without any issue.
I am using the same configuration used in the original experiments of the paper, which is based on older versions of the env packages. Here is the list of the versions I am using:
gpytorch 1.0.1
python 3.6.9
torch 1.8.1
torchvision 0.9.1I suggest you to replicate them on a conda environment. In particular, the gpytorch version may be the culprit here.
Hello I tried to run sine regression task DKT scripts as python train_DKT.py in sine folder and met the following issue. Environment gpytorch 1.6.0 python 3.9 torch 1.10.1
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py:38: NumericalWarning: A not p.d., added jitter of 1.0e-06 to the diagonal warnings.warn( /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py:38: NumericalWarning: A not p.d., added jitter of 1.0e-05 to the diagonal warnings.warn( /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py:38: NumericalWarning: A not p.d., added jitter of 1.0e-04 to the diagonal warnings.warn( Traceback (most recent call last): File "/Users/xichen.dxc/Desktop/project/gitlab/deep-kernel-transfer/sines/train_DKT.py", line 287, in
main()
File "/Users/xichen.dxc/Desktop/project/gitlab/deep-kernel-transfer/sines/train_DKT.py", line 183, in main
loss = -mll(predictions, gp.train_targets)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/module.py", line 30, in call
outputs = self.forward(*inputs, *kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/mlls/exact_marginal_log_likelihood.py", line 62, in forward
res = output.log_prob(target)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/distributions/multivariate_normal.py", line 169, in log_prob
inv_quad, logdet = covar.inv_quad_logdet(inv_quad_rhs=diff.unsqueeze(-1), logdet=True)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/lazy/lazy_tensor.py", line 1291, in inv_quad_logdet
cholesky = CholLazyTensor(TriangularLazyTensor(self.cholesky()))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/lazy/lazy_tensor.py", line 1004, in cholesky
chol = self._cholesky(upper=False)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/memoize.py", line 59, in g
return _add_to_cache(self, cache_name, method(self, args, *kwargs), args, kwargs_pkl=kwargs_pkl)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/lazy/lazy_tensor.py", line 435, in _cholesky
cholesky = psd_safe_cholesky(evaluated_mat, upper=upper).contiguous()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py", line 63, in psd_safe_cholesky
L = _psd_safe_cholesky(A, out=out, jitter=jitter, max_tries=max_tries)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py", line 45, in _psd_safe_cholesky
raise NotPSDError(f"Matrix not positive definite after repeatedly adding jitter up to {jitter_new:.1e}.")
gpytorch.utils.errors.NotPSDError: Matrix not positive definite after repeatedly adding jitter up to 1.0e-04.