search

ICSM / ampere

A tool to fit the SED and spectra of dusty objects to constrain, among other things, the dust properties and mineralogy
6 stars 2 forks source link

Optimised covariance matrices #29

Open pscicluna opened 4 years ago

pscicluna commented 4 years ago

For large datasets, the inversion of the covariance matrix and calculation of the likelihood will be a major bottleneck if we continue with the current naive (O(n^3)) implementation. This will become an issue when working with spectra with many thousands of data points, and will be particularly problematic for megapixel images, as well as spectro-interferometry and spectro-imaging.

There has been some recent work on general methods to solve these issues. Particularly useful to us could be the spleaf package (see the paper here), which implements optimised routines for a specific class of covariance matrices, those that can be defined as the sum of a semi-separable matrix (or a set of semi-separable components) and a symmetric LEAF matrix (a class of sparse, nearly diagonal matrices with the data stored in a leaf+branch approach to optimise memory footprint), to provide O(n) computing cost. We should attempt to use this S+LEAF method wherever possible, by using the spleaf package in ampere.

pscicluna commented 1 year ago

See also https://github.com/ICSM/ampere/issues/67. This is a bottleneck already for some datasets. We will likely need to consider ways of doing this lazily or in compiled code, perhaps building on Gpytorch (lazy kernel matrices) or JAX (easy compilation).