search

pytorch / pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration
https://pytorch.org
Other
80.5k stars 21.62k forks source link

[feature request] Multivariate normal CDF #29981

Open VincentBt opened 4 years ago

VincentBt commented 4 years ago

I'd like to discuss here the implementation of the multivariate normal cumulative distribution function (CDF), as the following code

from torch.distributions import MultivariateNormal
mvn = MultivariateNormal(torch.zeros(2), torch.eye(2))
mvn.cdf(torch.ones(3))

raises a NotImplementedError.

The cdf of the mvn has no closed-form solution, and is implemented in scipy (Fortran code) and in Matlab based on the work by Genz (paper here).

If needed, we can discuss the derivatives of the cdf w.r.t. the location and the correlation coefficient here. There exist formulas in the bivariate case but not in the general multivariate case (at least to my knowledge).

cc @vincentqb @fritzo @neerajprad @alicanb @vishwakftw

SebastienMarmin commented 4 years ago

Hello, Formulas in the general case for the derivatives w.r.t. location and covariance exists but they are quite complicated, and computationally expensive as they requires several numerical integrations. I derived them while ago (see last page of https://arxiv.org/pdf/1503.05509.pdf, there should be a earlier work on this but I didn't find). I made an attempt to bring this in PyTorch: https://github.com/SebastienMarmin/torch-mvnorm Any improvement on this code are welcome and will be well appreciated! Cheers

avidale commented 3 years ago

Is there any way to hope that Multivariate normal CDF will be implemented in Pytorch in the foreseeable future?

Balandat commented 2 years ago

@j-wilson put together an implementation for this (in python) with pretty impressive performance based on the method in https://dl.acm.org/doi/10.1007/s11222-014-9468-y. If there is enough interest we could consider upstreaming this (though since it's kind of loopy we may want to implement it in C++ at some point).

Is anyone aware of CUDA implementations of mvn cdfs?

jinchen-zhao commented 8 months ago

@j-wilson put together an implementation for this (in python) with pretty impressive performance based on the method in https://dl.acm.org/doi/10.1007/s11222-014-9468-y. If there is enough interest we could consider upstreaming this (though since it's kind of loopy we may want to implement it in C++ at some point).

Is anyone aware of CUDA implementations of mvn cdfs?

Hi Balandat, do you have a link for the implementation? I could not find that. Thanks!

Balandat commented 8 months ago

Yeah, it's part of BoTorch here: https://github.com/pytorch/botorch/blob/main/botorch/utils/probability/mvnxpb.py#L62