One GP for all the features

[jilsamia]

Hi, I am trying to duplicate this example: https://docs.gpytorch.ai/en/v1.1.1/examples/06_PyTorch_NN_Integration_DKL/Deep_Kernel_Learning_DenseNet_CIFAR_Tutorial.html but I would like to learn the same gaussian process for all the features unlike in the website (picture below). Does anyone know how I can do that ? Thanks ;D

[jacobrgardner]

This is more or less not possible with SKI, because the number of grid points would scale exponentially with the number of features you extract. If you want a GP on the full output dimensionality of the neural network, your best bet would be to use SVGP on top of a neural net instead.

[jilsamia]

Thank you @jacobrgardner for your reply, but my example above is also an SVGP on the top of a neural network (resnet101 on cifar10), the only issue I have is training the gaussian process on the whole dimensionality and getting one common GP for all the classes (instead of 10 GP). I used SVGP

[jacobrgardner]

Oh. If gp_layer is already an SVGP, then just don't do the transpose -- just pass some n x d features from the neural net to SVGP, and make sure SVGP has everything set appropriately (e.g., the inducing points should have dimensionality equal to the number of features extracted by the neural net).

[jilsamia]

Thank you for the explanation. I use this code to define my gaussian process knowing that the output of my neural network is N*10. I believe I have to change something here but I don't know exactly, it is my first time working with SVGP.

I defined my GP As:

The likelihood I used is: likelihood = gpytorch.likelihoods.SoftmaxLikelihood(num_features=model.num_dim, num_classes=num_classes) with num_dim=num_classes = 10 (the output of my NN has dimension N*10. If I only remove the transpose as you said, I got an error, which means I have to modify my GP, but I don't know what exactly.

Thank you.