search

STOR-i / GaussianProcesses.jl

A Julia package for Gaussian Processes
https://stor-i.github.io/GaussianProcesses.jl/latest/
Other
308 stars 53 forks source link

Weighted Regression Options #143

Closed nikpocuca closed 4 years ago

nikpocuca commented 4 years ago

It would be nice to have an option for having weights associated with each observation for weighted regression. I'm not sure how much work there would be for this, but you would have to have it come in through the log density, as in have it weighted by the log density here:

log_dens(gp.lik,F,gp.y)

something like:

weights .* log_dens(gp.lik,F,gp.y)

and all other mentions of the log likelihood function.

Otherwise weights are just a constant 1.0 ?

chris-nemeth commented 4 years ago

Thanks for raising this. It's an interesting idea. We'll take a look at how much code will need to be changed.

nikpocuca commented 4 years ago

In the interest of time, it may be interesting to just create a weighted kernel in the form of 

k(x_i,x_j) = w_i * w_j * k'(x_i,x_j)

where k'(x_i,x_j) can be any kernel of choice, say squared exponential as simple? The weights for my problem are constant so I would just pass them in as a vector.

chris-nemeth commented 4 years ago

Interesting idea. Could you create a PR?

maximerischard commented 4 years ago

It's not so clear to me how to think of the weights from a probabilistic point of view. In particular, your two proposals are actually very different.

Your first proposal, if I understand it correctly, scales the likelihood. In the normal case, that's equivalent to scaling the variance of the individual observations, which is covered by issue #7 already.

In your second proposal, I think that's equivalent to scaling up the Gaussian process by a different constant at each observation. It's a slightly odd thing to do, and would be better accomplished by a custom kernel. One would also need to specify this constant at each prediction.

@nikpocuca if you want to pursue this further, could you link to implementations in other Gaussian process packages (in julia or other languages) to get a better understanding of how people would be using this feature, and which approach they've taken? I'll close the issue for now but please feel free to reopen.