ARPACK fails to converge when calculating eigenvalues

[kyleabeauchamp]

We sometimes see something like this:

scipy.sparse.linalg.eigen.arpack.arpack.ArpackNoConvergence: ARPACK error -1: No convergence (100001 iterations, 0/5 eigenvectors converged) [ARPACK error -14: DNAUPD did not find any eigenvalues to sufficient accuracy.]

The question is:

1. How should we deal with this?
2. Is there any bug on our end?
3. Can we adjust ARPACK to better avoid this?

[rmcgibbo]

There aren't that many new options to try.

One option to try is shift-invert mode. I believe this is only available in newer scipys thought. The idea is that it lets you look for eigenvalues close to a certain value. I think it's most useful for finding eigenvalues that are not the smallest or largest, so I doubt it's going to be too helpful.

Another option is to try seeding with some guess at the eigenvector. Perhaps running the hermetian eigensolver on the T+T.transpose() could give something useful. This is a pure guess.

The bigger issue is that we're basically flying blind here. Maybe if we should detect errors like this, and, if the matrix is less than a couple of tens of megabytes, save it to disk. That way the user could do more in-depth debugging if they want to?

Do you know what size your matrix is? Perhaps we should increase the dense cutoff, which is currently at 50.

[kyleabeauchamp]

I'm still waiting on the info about the matrix.

[kyleabeauchamp]

Another idea is to use the sparse hermitian generalized eigensolver. The idea is that

Tx = e x

is equivalent to (I think)

C x = e D x

D is a diagonal matrix of counts, and C is a symmetric matrix. Thus, we can use a symmetric generalized eigensolver, which might be faster and more robust than a non-symmetric solver.

[rmcgibbo]

Yeah, if the math works that could be a viable option. I'm not really familiar with numerical methods for generalized eigenproblems, but I think that in general Hermetian problems have way better numerical properties.

[kyleabeauchamp]

So it's really as simple as passing along the matrix M in our ARPACK calls.

[rmcgibbo]

I just did a simple benchmark, and I think we might want to up the dense_cutoff to ~150.

scroll down to the bottom for the result: http://nbviewer.ipython.org/4541776/

[kyleabeauchamp]

This might also be solved by #322, closing for now.