Correct handy energy, but issues with the timing

[pkozlows]

My timings are looking like this: full matrix generation: 0.9805660247802734 Once I have this matrix, generating the eigensystem using numpy: 0.08746600151062012 Same, but generating the eigensystem with the Davidson algorithm: 0.015346050262451172 The time for getting a correct energy with my handy implementation depends on the tolerance that I am asking for: for tol=1e-6, I got 2.2760820388793945, but when I ask only for a tolerance of 1e-4, it is 1.3750641345977783. Upon first glance, it is strange that the matrix generation is taking so little time. Maybe this is because I am dealing with too small of a matrix? This seems like a tough time to beat, even for handy if I optimize further. All of my implementations are giving a result of something like: -7.8399.

And then I have a question about the Davidson algorithm. I noticed that when I add the perturbation to my code, like I do at the end of my line here, it seems to accelerate my code considerably.

guess_space = np.eye(dimension, begin_search_size) + .000001

GPT-4 had this to say, which doesn't seem so much of a hallucination, but I don't completely understand it. Would you be able to explain a little more?

Acceleration: As you've observed, in some cases, adding this small perturbation can help the Davidson algorithm converge faster. This can be attributed to the initial search space being more "representative" or "spread out" over the eigenspace of the matrix.

[Walter-Feng]

If you are taking more time in KH algorithm than matrix generation, it is likely that you have some significant space for optimization in your code - it should be that KH algorithm be faster, Garnet does care about this. Nevertheless, I would imagine Garnet would be happy about your success in these two implementations if you tell him your progress at this stage.

Before exploring the reason, let me confirm that initial guess is important in quantum chemistry. A good initial guess can give, for example, 1/2 -> 1/4 of the original computational time. In worst cases, like DMRG or some other complicated post HF methods, a bad initial guess can lead to wrong, or even no convergence of the electronic state.

If I recall the Davidson correctly, you start with some initial vectors whose number is smaller than the total dimension ( otherwise how can you be having faster diagonalization right? ...), then you can think of the following,

if are looking for a specific point in a three dimension, say (1,2,0.1), and you only provide (1,0,0) and (0,1,0) as initial vector, you kind of imagine it would be hard to get the 0.1 in the z dimension. but if you add a little bit component of z in the initial vectors, it does not have to hover around the x-y plane and think about what is missing.

[pkozlows]

Was a good explanation about Davidson. I can confirm that you are better than GPT-4 :)

It does seem very fishy to me that my matrix generation is taking so little time: matrix generation time: 0.9470911026000977

I recognize that timing may be very machine dependent, so I should take the independent result with a grain of salt. If you were in my shoes, would you be worried about this timing? I am getting a correct lowest energy and am passing all of my unit tests. Is there anything else that I should be expecting from the matrix generation? Or would you spend your efforts trying to optimize the handy implementation further?

[Walter-Feng]

I wouldn't bother too much, maybe just see whether Garnet wants a better implementation or not - I would bet he does not. It might be more important if you could explore a bit more about theoretical chemistry, either electronic structure or dynamics. Maybe the professors' work, maybe some hot topics, maybe some sophisticated code, maybe some short-term projects.

I can only help you with technical terms, or maybe some tips for writing better code, but you are at the stage where you are applying for grad school and, the clock is ticking. Regarding this, Garnet will be a better person to offer you advice on how to continue with research / learning and make you more competitive.

[pkozlows]

this is good advice. I want to apply to the NSF graduate research fellowship program, but the application involves having a research proposal, and I'm kind of lost on this part.