gem / oq-engine

OpenQuake's Engine for Seismic Hazard and Risk Analysis
https://github.com/gem/oq-engine/#openquake-engine
GNU Affero General Public License v3.0
373 stars 272 forks source link

Fixed the issue of small negative eigenvalues in conditioned GMFs #9784

Closed micheles closed 2 weeks ago

micheles commented 2 weeks ago

By introducing a correlation cutoff. Also changed the method of multivariate_normal to cholesky since the numpy docs say

multivariate_normal internally uses linear algebra routines, and thus results
may not be identical (even up to precision) across architectures, OSes, or even builds.
For example, this is likely if cov has multiple equal singular values and method is 'svd'
(default). In this case, method='cholesky' may be more robust.

This is critical because the results of conditioned GMFs were machine-dependent, i.e. not reproducible at all. See the huge change in openquake/qa_tests_data/scenario_risk/conditioned/expected/aggrisk.csv. The aggrisk is now stable even if I change the cutoff:

$ oq run job.ini -p correlation_cutoff=1E-11 && oq show aggrisk
$ oq run job.ini -p correlation_cutoff=1E-12 && oq show aggrisk
$ oq run job.ini -p correlation_cutoff=1E-13 && oq show aggrisk

gives always the same results within the precision of the view.

raoanirudh commented 2 weeks ago

If we need to switch to cholesky, there are existing helper functions in conditioned_gmfs.py that should help with ensuring that the covariance matrices are positive (semi-) definite, eg.: https://github.com/gem/oq-engine/blob/engine-3.20/openquake/hazardlib/calc/conditioned_gmfs.py#L835-L901

(copied from statsmodels.stats.correlation_tools.cov_nearest)