Open mekty2012 opened 2 years ago
While I was testing this bug, I found that this happens when we have too many data points, without multi dimensional input.
train_x = torch.cartesian_prod(torch.linspace(-1, 1, 5), torch.linspace(-1, 1, 5))
, the code works well.
import torch
import gpytorch
train_x = torch.linspace(-1, 1, 500)
train_y = torch.stack([ torch.sin(train_x (2 math.pi)) torch.sin(train_x (2 math.pi)) + torch.randn(train_x.size()) 0.2, torch.sin(train_x (2 math.pi)) torch.cos(train_x (2 math.pi)) + torch.randn(train_x.size()) 0.2, torch.cos(train_x (2 math.pi)) torch.sin(train_x (2 math.pi)) + torch.randn(train_x.size()) 0.2, torch.cos(train_x (2 math.pi)) torch.cos(train_x (2 math.pi)) + torch.randn(train_x.size()) 0.2, ], -1)
class MultitaskGPModel(gpytorch.models.ExactGP): def init(self, train_x, train_y, likelihood): super(MultitaskGPModel, self).init(train_x, train_y, likelihood) self.mean_module = gpytorch.means.MultitaskMean( gpytorch.means.ConstantMean(), num_tasks=4 ) self.covar_module = gpytorch.kernels.MultitaskKernel( gpytorch.kernels.GridInterpolationKernel( gpytorch.kernels.RBFKernel(), grid_size=10, num_dims=1 ), num_tasks=4, rank=0 )
def forward(self, x):
mean_x = self.mean_module(x)
covar_x = self.covar_module(x)
return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)
likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=4) model = MultitaskGPModel(train_x, train_y, likelihood)
model.train() likelihood.train()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1) # Includes GaussianLikelihood parameters
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)
for i in range(training_iterations): optimizer.zero_grad() output = model(train_x) loss = -mll(output, train_y) loss.backward() print('Iter %d/%d - Loss: %.3f' % (i + 1, training_iterations, loss.item())) optimizer.step()
So I'll fix the title of issue from multi-dimensional to large.
I just encountered the same problem with 2-dim input and 2-dim output for 5000 training points. I got the grid size from gpytorch.utils.grid.choose_grid_size(inputs_trn, 1.0)
and created the covar_module
like this
base_kernel = gpytorch.kernels.MaternKernel()
kernel = gpytorch.kernels.GridInterpolationKernel(kernel, grid_size, num_dims=num_tasks)
covar_module = gpytorch.kernels.MultitaskKernel(base_kernel, num_tasks, kernel_rank)
Yeah - this looks like a bug in GPyTorch/LinearOperator. I do not have the bandwidth to investigate this at the moment, so I'm looking for someone who can put up a PR to fix this.
I took a look into this error and found the root cause for the type error is under KroneckerProductAddedDiagLinearOperator._solve in LinearOperator. On line 177-181, we try to convert the lt and dlt matrices to torch.float64.
# again we perform the solve in double precision for numerical stability issues
# TODO: Use fp64 registry once #1213 is addressed
rhs = rhs.to(symeig_dtype)
lt = self.linear_op.to(symeig_dtype)
dlt = self.diag_tensor.to(symeig_dtype)
But this operation could be bad for InterpolatedLinearOperator as it converts the indices matrices of the operator (left_interp_indices, right_interp_indices) to float64 too, and this would further cause a type error when we try to create a sparse matrix using the indices matrix.
As for the input size dependence of this type error, we can avoid getting this type error if we call Cholesky for the inverse quadratic term and logdet term in _linear_operator.inv_quad_logdet(), when settings.fast_computations.log_prob.off() == true
or input_size * output_dimension <= settings.max_cholesky_size.value()
, which is 800.
I want to collect some suggestions on what is the best way to fix this issue from people with more experience on this package. One straight-forward way is to avoid converting the index matrices to torch.float when calling to() in _linear_operator, but there might be some other easier way to fix this problem.
I want to collect some suggestions on what is the best way to fix this issue from people with more experience on this package. One straight-forward way is to avoid converting the index matrices to torch.float when calling to() in _linear_operator, but there might be some other easier way to fix this problem.
Yeah it seems like not converting the index matrices in InterpolatedLinearOperator
when calling to()
is something that we'd want anyway. We can basically just overwrite the to()
method on InterpolatedLinearOperator
to only move the base linear operator.
Would you be willing to help out and put up a PR for this on the linear_operator
repo?
Yeah it seems like not converting the index matrices in
InterpolatedLinearOperator
when callingto()
is something that we'd want anyway. We can basically just overwrite theto()
method onInterpolatedLinearOperator
to only move the base linear operator.Would you be willing to help out and put up a PR for this on the
linear_operator
repo?
Sure, I will work on it and add an unit test for that.
🐛 Bug
I was trying to extend MultiTaskKernel's example by including GridInterpolationKernel, with the input being more than one dimension. However when I'm using all three features (multitask, grid interpolation kernel, multi-dimensional input), I get error in internal of library.
To reproduce
Code snippet to reproduce
Stack trace/error message
Expected Behavior
Very similar implementation with batch multitask works well. So I'll add its implementation for the comparison and some visualizations of data.
Visualization on each tasks![output](https://user-images.githubusercontent.com/41375658/160559011-6400f725-482c-4c88-8894-5b490b0b8ecd.png)
System information
Please complete the following information: