investigate optimisation

[LucaYoy]

• [x] change hamiltonian
• [x] Random vs close to Id (model C)
• [x] GD vs Polar (using close to Id, model C)
• [x] Compare models
• [x] Vary N (model C)
• [x] Vary M (model C)
• [x] tune learning rate

[LucaYoy]

@as2457 @NicholasSynesi

help! I am receiving this error on HPC when trying to run my script. I have memory put to 10g on the sh file and then I put it to 20g, but it still kills my jobs... @NicholasSynesi what did you put your mem to ?

[AdamSmith-physics]

@Luca, did what N are you using?

Did you try running a script on your laptop to see how much RAM it is using as a test? It might just take a lot...

Can you also send a screenshot of the shell script? Did you make sure that you have uploaded the latest version to the HPC?

[LucaYoy]

@as2457 I'm using N=14. I have run it on my PC for N=6 and everything seemed fine, I'm not sure how to check RAM on my pc plus how do I check for N=14? Or do I check for 6 and then scale that exponentially to 14?

[AdamSmith-physics]

@LucaYoy Just try run it for N=14 on your laptop. You can try building up with N=10, 12, 14 (in case you run into memory issues on your laptop). While the script is running, you can check the memory usage in activity monitor. Se the number of iterations high enough that you have time to check.

[LucaYoy]

@as2457 @NicholasSynesi N = 14, M = 5 , sweeps = 10000. Not sure how to read this, im not sure which memory section is the one relevant for HPC

Update: Ramped up memory to 40GB still getting my jobs killed

[AdamSmith-physics]

@LucaYoy something doesn't look quite right. Can you send a screenshot of the shell script as well?

[LucaYoy]

@as2457

The script is killed when it reaches the approxSateRand line (line 19)

[AdamSmith-physics]

@LucaYoy @NicholasSynesi if you make sure GitHub is up to date and let me know which are the files to run on HPC I'll take a look tomorrow morning to figure it out.

[LucaYoy]

@as2457 everything is pushed on the 43-investigate-optimisation branch, the HPC scripts are placed in HPC folder and im planning to run the main.py files using the sh script. Have a nice rest of weekend!

[AdamSmith-physics]

@LucaYoy @NicholasSynesi I found out the issue and pushed the fix to the branch.

Basically, there were two things:

• You were using sum and reduce to construct the matrix. While this is elegant, the way it actually works is it creates each of the large matrices and then sums them together at the end. Since I find the final Hamiltonian takes up ~5gb, this meant you had to store ~ 14 x 5gb, which is far too big!
• Second, I realise I never told you that you can use sparse matrices since the Hamiltonian is actually mostly zeros. When using sparse matrices you can then use eigsh to get only the lowest eigenvalue without computing the whole spectrum. This reduces memory to about 100mb, and the state is generated in under a second.

Note:

• I added the new Ed functions in ExactDiag_adam
• I modified the main_RandomVsCloseToId.py to include a function that tracks memory usage.
• I changed which to which-1 for 0 indexing
• I reduced number of iterations to 200 so I could get it to complete in a few seconds on my computer (you should change this for the HPC)
• It now uses ExactDiagSparse
• I added several print statements to keep track of where we are up to

I think you can now try on HPC, and probably get away with 1G (to be safe), or even as low as 200M RAM.

[NicholasSynesi]

Just to make a note of my findings: Everything on my end seems to be fine, apart from the fact that I highly underestimated how long the scripts would take for additional layers. When I was running a few scripts in the meeting, it was for 3 layers. Pushing it up to 5 layers, even for 8 qubits the rates I'm getting for 8 qubits are 1000 iterations per 30 seconds, which is of course fine.

I managed to get 10 qubits 5 layers 20,000 sweeps working which ran overnight. However for 14 qubits I don't think it's feasible. Doing a test on my laptop for n=14 m=5, 10 sweeps (yes, 10) has just taken 12 minutes, so it would take about 2 and a half weeks for this to run! I'm going to assume to push the number of qubits down to 12, which does seem doable (on my laptop I'm predicting 12 hours per 20,000 sweeps), and keep everything else the same as discussed in the meeting.

[AdamSmith-physics]

@NicholasSynesi yes, it sounds sensible to start with N=12. You can then always try push to N=14 once you have these results.

Note, I don't think the scaling will be as bad as you think. For 10 sweeps I am sure most of that 12 minutes will be setting up the Hamiltonian and finding the ground state. Try this again with the sparse ExactDiagSparse function I added to Luca's code. I was able to run 400 iterations in a few seconds.

[AdamSmith-physics]

@LucaYoy I found another issue with your code. Don't use param.values(). Since all of the variables except run is the same for all, just set them statically. E.g.

N = 14 j = 1 h = 1 g = 1 M = 3 maxIterations = 200 run = which + 1

If the variable does depend on the job in the array, then set it with N = param["N"]

[NicholasSynesi]

Oh yeah the new exact diagonalization method was definitely what was taking the most time, I should be able to get 14 qubits working now.

[LucaYoy]

@as2457

• why which-1 , the sh file already has array 0-9 so 0 indexed
• is param.values() also memory intensive just wondering
• when I run can I run the shell file multiple times with the different main files and change the array in the sh file so that everything runs in parallel?

[AdamSmith-physics]

@LucaYoy sorry you are right. You can just use which. My bad.

param.values() resulted in strange behaviour when I ran it. I don't think you can guarantee that a dictionary is ordered, so you should read them out using keys. Otherwise just use a list where you know the order of elements. Since you only change one thing at a time, I would just assign your variables individually.