WIP: reduce the memory use of EAKF for large networks

[jinlong83]

This pull request aims at reducing the memory use of EAKF. Two things have been done:

• use la.null_space() to get the full singular vectors in regularized SVD;
• update the implementation of calculating Sigma_u in regularized SVD.

[odunbar]

I came across this, it's old so not sure if still true, but seems to indicate that there's a sparse SVD that's more memory efficient. scipy.sparse.svds

http://fa.bianp.net/blog/2012/singular-value-decomposition-in-scipy/

a quick test in python

import numpy as np
import scipy.sparse.linalg as spla
Z=numpy.random.random([20000,100])
u, s, _ = spla.svds(Z,k=99,return_singular_values='u')

is quick while

import numpy as np
import scipy.linalg as la
Z=numpy.random.random([20000,100])
u, s, _ = la.svd(Z)

takes a long time and uses a lot of my laptop memory.

It should be noted that the sparse version doesn't have like full_matrices flags etc. so these may still need to be computed

[dburov190]

scipy.linalg.null_space seems to be using SVD itself, is this okay? https://github.com/scipy/scipy/blob/v1.5.1/scipy/linalg/decomp_svd.py#L330-L388

[odunbar]

We were also using SVD directly to do this before. But the null_space function seemed like a safer option (even though it's a thin wrapper) as it is likely tested. I figured that they hadn't found a quicker way to build up the space.

[odunbar]

Update:

  cov_inv = np.linalg.inv(cov)

This is what kills us! do np.diag(1/np.diag(cov))

This along with the sparse svd should do the trick i think.

sparse svd doesn't have this flag full_matrices so in the case of a method using F_full,G_full we will need another way of computing the orthogonal complement of the singular vector matrix. However Jinlongs method only works with F and G initially, so we might get away with not needing to calculate these either.

[odunbar]

We don't need to calculate Sigma any more either - as we don't explicitly do Sigma_u = A . Sigma . A.T