Setting of tol after except RuntimeError in MultivariateNormalQMCEngine

[linjing-lab]

🚀 Setting of tol after except RuntimeError in MultivariateNormalQMCEngine

Motivation

Is your feature request related to a problem? Please describe.

I have some understanding of solving matrix eigenvalues, and I found tol = 1e-8 if eigval.dtype == torch.double else 1e-6 located at MultivariateNormalQMCEngine after torch.linalg.cholesky(cov).transpose(-1, -2) except RuntimeError, but I was wondering how to determine "Covariance matrix not PSD." by distinguishing the tol value through torch.double when torch.any(eigval < -tol) is enabled, and how to fixed the value of tol as 1e-8 when eigval.dtype is torch.double.

Pitch

Describe the solution you'd like

I think studying the floating decimal representation range of torch.double and the solving precision of the torch.linalg.eigh method to determine the value of tol. If only the solution paths and floating 0 of negative field distinguish the type of matrix, then setting tol = 1e-8 is not a way to ensure convergence.

Describe alternatives you've considered

I haven't found an accurate method for determining positive semi-definite matrix, but reducing the tol value to less than 1e-8 would be more accurate detection, so same as tol=1e-6 when eigval.dtype is torch.float32.

Are you willing to open a pull request? (See CONTRIBUTING) I don't think submitting a pull request would benefit this approach.

Additional context

[Balandat]

I am not sure I understand the question/proposal - are you saying that the cutoff thresholds of 1e-8 and 1e-6 are not the best / most meaningful? That may well be. The difficulty is also that it really depends on how poorly conditioned the matrix is for how precisely the evals can be estimated.

The point of this logic here in the context of Bayesian Optimization is not necessarily to have the sharpest possible detection / highest possible resolution of determining whether the matrix is psd or not, but rather a way of raising an error when the matrix is either clearly not psd or extremely poorly conditioned and otherwise provide reasonable samples from the distribution.

[linjing-lab]

OK, I propose this issue because I discovered an alternative way to determine whether a matrix is positive definite when improving the operator h2h by changing (all(values) > 0) & (np.linalg.cond(hessian) <= m) to matrix_rank wrapped by numpy, so matrix_rank is an accurate method for me to make a hessian matrix as a positive define matrix. That's why I mentioned that 'I haven't found an accurate method for determining positive semi-definite matrix' and 'floating 0 of negative field', I improved the cutoff precision of correcting Hessian matrix by setting np.linalg.matrix_rank(hessian) in optimtool-2.4.2.

[Balandat]

Thanks. I don't know if this really helps us here - the fact that the Cholesky decomposition fails already tells us that the covariance is (numerically) not positive definite, so it's not clear what np.linalg.matrix_rank would add here. This is really a design choice about how the tradeoff between tolerant we want to be of ill-conditioned matrices and making sure that we raise errors if the matrix is indeed not (mathematically) positive semidefinite.