is it a good idea to pursue direct CI before getting a correct FCI energy?

[pkozlows]

I recently read about how a determinant can be decomposed into beta/alpha strings. I think the method is called "direct CI." I am currently doing a lot of manual hacking in order to try to get a correct energy, but to no avail yet. After having read about how a determinant can be decomposed into spin strings, this is very hard to unsee, as this method just seems so much simpler. Did you implement "direct configuration interaction" into your program? Is it even a good idea to start pursuing this before I get a correct energy? I could be very wrong, but I'm feeling like I'm understanding the spin component of FCI decently now. I feel like if I were to pursue direct configuration interaction right now, this would be a more efficient means of getting the correct energy. Do you have any thoughts on this?

[pkozlows]

The first roadblock that I see is in ordering the determined bases, as I was able to do when I didn't decompose my bases into spin strings. Basically, I don't know how to decompose this line into a spin string implementation. https://github.com/pkozlows/fci/blob/7d853fc0ba1436646301d8a387fe38170c222844/%20main.py#L35 Where I sorted by: https://github.com/pkozlows/fci/blob/7d853fc0ba1436646301d8a387fe38170c222844/%20main.py#L18 this implementation is conceptually simpler for me to think about, but I don't think it is so efficient. Is the lexical order of the way the spin strings are generated by itertools how I want to leave them moving forward, or should I make a different ordering? https://github.com/pkozlows/fci/blob/7d853fc0ba1436646301d8a387fe38170c222844/%20main.py#L25

[pkozlows]

Never mind. I have an idea of how to order the bases with the direct implementation Next time. I should think before I message. 🙂 I guess the one question I still have for you is whether I will lose something by moving to the direct implementation before getting a correct energy.

[Walter-Feng]

Then would you mind explaining what direct CI is? I guess I failed to understand what direct CI is when I was implementing full CI.

Generally yes, it benefits both of us if you can find out the solution yourself, and of course, finding out the solution takes time. Were my instant replies spoiling you too much? :p

Regardless of direct CI (or whether I implemented direct CI), you would actually implement two versions of Full CI, one with Full-CI matrix, the other using Knowles-Handy algorithm ( which I don't think is anything related to direct CI). What I felt was, KH algorithm does not necessarily require correct implementation of Full-CI matrix. Nevertheless, the basis, the creation/annihilation operators, and the Davidson diagonalization you haven't touched yet, will still be crucial to both implementations. As it is not merely about "implementing Full CI", but also with "understanding all the relevant topics behind Full CI", I would still insist getting correct energy. If direct CI is related to Full-CI matrix build, I will have no negative coment if you choose direct CI, if it makes things simpler.

Now I am pasting a link to my solution for Full CI matrix, in sparse matrix form. You can have a peek if you still fail to identify the problem in a few days.

[pkozlows]

So, my understanding is that direct CI is separating the whole determinant into its constituting spin strings, which you probably did from the beginning, but I didn't do that at first.

I am currently getting stumped by resolving the phase factor for one-electron difference. Specifically, I am having trouble getting the linked assert statement to pass. When I think about bubble sorting the whole determinant, it makes sense that the phase factor would be -1, but when I consider each individual spin string, there are only two swaps required to bring the only alpha difference spin orb to the front. I can't just sort {4, 0, 2} from {0, 2, 4} and {6, 0, 2} from {0, 2, 6} and call it a day because this is omitting the relevant information that 4 < 5 < 6 (where 5 is from the beta ors {1, 3, 5}). I'm not sure how to represent this fact. https://github.com/pkozlows/fci/blob/1ec6ed8075a8cb9ad31c1d282af272d724881d54/face_factor.py#L43 I am fully aware that I was spoiled by your instant replies, but I will say that having some help at the start of the project helped to motivate me to make some progress (more coming hopefully). So, thank you a lot :) I will try my best to continue this project as independently as possible from now on, but it looks like I am not off to a great start by virtue of asking this question lol

[Walter-Feng]

It will be a whole mess if you align the operators first by its orbital indices. Are you implementing as

0 1 1 2 4 .... a a b b a

It saved me a lot of time by splittting into alpha section and beta section,

0 1 4 .... 1 2 ... a a a .... b b ...

Then moving the beta operators before alpha when calculating phase factor is equivalent to moving to the head of beta strings. Or in other words, it only requires you to sort in alpha section and beta section independently.

[pkozlows]

so now, I have the following implementation: the format is: ((bra_alpha, bra_beta), (ket_alpha, ket_beta)) so, I am understanding that the spin orbs: (({0,2,4}, {1,3,5}), ({0,2,6}, {1,3,5})) can and ought to be simplified to a special orb representation of: (({0,1,2},{0,1,2}), ({0,1,3},{0,1,2})) which based of the ordering of the sets in the tuples, encodes all of the necessary information i.e. it is possible to recreate the entire spin orb determinants: ({0,1,2,3,4,5},{0,1,2,3,5,6}) from the previous. so, I believe, I am doing what you mentioned here:

It saved me a lot of time by splittting into alpha section and beta section

I am still getting confused by this part:

Then moving the beta operators before alpha when calculating phase factor is equivalent to moving to the head of beta strings. Or in other words, it only requires you to sort in alpha section and beta section independently.

specifically, when you say sort the alpha section and beta section independently, do you mean sorting that section for both dets together? I am probably making no sense here, but I don't know how else I would phrase it? I am not understanding how this sorting would come about.

[pkozlows]

({0,1,2,3,4,5},{0,1,2,3,5,6}) should be giving a face factor of -1. however, if I am sorting (({0,1,2},{0,1,2}), ({0,1,3},{0,1,2})) to (({2,0,1},{0,1,2}), ({3,0,1},{0,1,2})) this will only take 2 (bra_alpha) + 2 (ket_alpha) = 4 swaps, returning a face factor of 1. I guess this gets to the crux of my question.

[Walter-Feng]

I understand my English may not be enough to elucidate my mathematical thinking.

When you mention

({0,1,2,3,4,5},{0,1,2,3,5,6}) should be giving a face factor of -1

I would guess you are calculating phase factor by assuming the order of spins as (alpha, beta, alpha, ....) for determinants ( 4<->5 ). Otherwise it's not clear to me how you calculated the phase factor.

If you are doing what I did, from your words "so, I believe, I am doing what you mentioned here", then you should have a phase factor of 1. You should not be calculating in the way similar to "({0,1,2,3,4,5},{0,1,2,3,5,6})".

[pkozlows]

I don't think your english was so much of a block; I was just not being clear enough. I think I understand what I need to do now, so thanks.

[pkozlows]

I hope this will be a quick one, but I changed to a direct implementation that seps alpha and beta strings, and I am kidding a very close energy to the correct one, but not quite still. now, I an getting: -7.833036080146565 compared to the correct one of: -7.8399080148963369. I was taking a look at your code, and it seems to have very similar functionality to what I wrote. I have been thinking about what you said a few weeks ago a lot recently.

Yeah it looks closer, I would guess your phase factor is done correct, and you have treated most of the excitations correct, maybe one or two case where you did not have all the terms or you added something extra. You may want to check two-electron excitation cases more carefully, I would guess there is one term scientifically wrong in one of the cases, considering the magnitude of the error.

I know you can't say anything for certain, but I just want to run a few cases by you. should I be able to safely assume that I am doing the face factor correctly? I have poured over the two electron excitation cases allot today, and they look fine to me. I assume that the problem would come from them though? I completely understand that I should be trying to troubleshoot this on my own, but I was just thinking I would throw this out there. I will keep locking over this for the next few days, and if I get an idea before you respond, I'll let you know :)

[Walter-Feng]

The only way you can feel reassured about your code is to have enough unit tests that covers the typical cases.

The result looks close, but not quite there, has two possibilities - one is the error is relatively small compared to the whole system. That is the main reason I was guessing you have something wrong in 2-electron excitation. The other is the error cancels out such that the result is close, but not necessarily the error itself is small. This can be a case if phase factor is implemented in a wrong way, as the error can average among wrong terms. Or it can be there are multiple bugs, but averaged in the result.

In electronic structure, an error of 1e-3 level cannot be regarded as the same number. There should be non-trivial error.