bigmemory (or similar method) for large matrix storage

NIEHS / PrestoGP

Penalized Regression on Spatiotemporal Outcomes using Gaussian Processes a.k.a. PrestoGP

https://niehs.github.io/PrestoGP/

0 stars 0 forks source link

bigmemory (or similar method) for large matrix storage #28

Open ericbair-sciome opened 7 months ago

ericbair-sciome commented 7 months ago

Utilizing the correlation based distance can require storage of the full covariance matrix. We will explore ways to ensure it is scalable for n > 60K.

https://fbertran.github.io/bigalgebra/

kyle-messier commented 7 months ago

a def v2.0

kyle-messier commented 1 month ago

@ericbair-sciome Another option to consider for 2.0 is Apache-Arrow. Fast and efficient read/write combined with adapting how the Vecchia neighborhood is stored and updated could help.

kyle-messier commented 1 month ago

We'll want to analyze areas in PrestoGP where memory is built up, opportunities where off-loading memory in exchange for I/O could be used.