[Feature Request] Hierarchical priors?

[fonnesbeck]

🚀 Feature Request

It would be useful to be able to set lengthscale (and other kernel parameter) priors to be hierarchical, so that a parameter values among individuals in a population could be informed partially through others in the population.

Motivation

I'm currently using IndependentModelList to batch fit sets of related GPs, which are independent but should a priori have kernel parameters that are similar. Some of the GPs with sparser data are not very stable, and would benefit from a hierarchical setup.

[jacobrgardner]

So if you want the models to be correlated rather than independent, it sounds to me like what you are really wanting to do is use a multitask GP, which is something we have support for a few different styles of. Have you looked in to any of these?

[fonnesbeck]

Yes, I did explore these early on, but if I understand them correctly the kernel hyperparameters are the same for all outcomes, which is probably unlikely for the system I am modeling. Hence, my question about hierarchical parameters.

I assume, then, that its not possible to specify a distribution for lengthscale_prior that has learnable parameters that could be given a common distribution to constrain their values?

UPDATE: tried a multitask implementation just now, but an additional issue is that there are a large number (>300) of tasks to be fit, and I quickly run out of memory (tries to allocate 30 GB) when attempting to run it on the GPU. I suppose a coregionalization kernel might work, but I don't see it implemented.

[fonnesbeck]

I noticed #700 so I assume there is no coregionalization kernel currently.

[jacobrgardner]

To your first question: I believe you should be able to put torch.nn.Parameters in a prior and have them be learned -- since those parameters will get used in computing the log prior term, they should get learned

For multitask GPs, There is both lcm_kernel and multitask_kernel. See https://github.com/cornellius-gp/gpytorch/tree/master/examples/03_Multitask_GP_Regression for example multitask GP usage. We don't explicitly implement linearly mixing the function values at the end because this is relatively trivial to do with standard pytorch+gpytorch: just register a weight matrix W and modify the return of forward.

[caozhen-bupt]

Is there a way to do hierarchical priors now? I would like to set, for example, a lengthscale prior l~LogNormal(x, y), where x follows a distribution, say, Normal(0,1), and y also follows a distribution, Gamma(1,1). I wonder whether you have some suggestions. Thanks!

[gpleiss]

At the moment, it's not available. Our team is a bit stretched, so we welcome a PR for this feature.