Documentation about distributed calculations

[TerraVenil]

Hi, @rrmeister , @TysonRayJones. I was working on configuration distributed calculations with pyQuEST and would like to share a few cases that appeared during setup. Would be glad to get feedback and if need will create separate tickets.

First of all I have started an investigation from the main QuEST repo looking for examples and documentation on how to set up distributed calculations in a simple app. Or I was so unlucky or information about configuration of distributed calculation was so limited that I found only reference to original article published in Scientific Report QuEST and High Performance Simulation of Quantum Computers and report with benchmarks, but it's not what I was looking for. Also I was looking into the pyQuEST package but probably because it's so new it's too early to add such an example. Anyway, I guess one small page can help decrease the entry barrier for newcomers and also demonstrates a real example of using distributed calculations with a small application similar to https://github.com/QuEST-Kit/QuEST/tree/master/examples. So, to start with something I come up with a small example:

import pyquest
from pyquest import Register, Circuit
from pyquest.unitaries import H
from mpi4py import MPI

def main() -> int:
    mpi_comm = MPI.COMM_WORLD
    mpi_rank = mpi_comm.Get_rank()
    mpi_size = mpi_comm.Get_size()

    print(pyquest.env)
    print(f"Rank: {mpi_rank}, Size: {mpi_size} running on {MPI.Get_processor_name()}")

    num_qubits = 16
    reg = Register(num_qubits)
    h_gates = [H(i) for i in range(num_qubits)]
    circ = Circuit(h_gates)
    reg.apply_circuit(circ)

    result = reg.prob_of_all_outcomes([i for i in range(num_qubits)])

    if mpi_rank == 0:
        print(result)

    return 0

if __name__ == "__main__":
    import sys

    sys.exit(main())

And indeed pyQuEST(QuEST) just works! but really you have to spent some time and it's not obvious from the first look how it just works.

Challenges have started to appear at the beginning, as always 🤓 I could't built pyQuEST for Python 3.10 because wheel for numpy==1.20.1 was absent in PyPi https://pypi.org/project/numpy/1.20.1/#files. I was so naive that decided to override it with numpy 2.0.0:

ValueError: Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.
File "/main.py", line 35, in <module>
sys.exit(main())
File "/main.py", line 24, in main
result = reg.prob_of_all_outcomes([i for i in range(num_qubits)])
File "core.pyx", line 581, in pyquest.core.Register.prob_of_all_outcomes

after which I had to take the last compatible version numpy==1.26.4 that has the wheel for Python 3.10. So I am curious how pyQuEST was built for Python 3.10 if numpy wheel was missing? And suggestion, is it possible to update build requirements to latest numpy version 1.xx or am I missing something?

Next "challenge" was to understand importance of the following message

ERROR: Trying to initialize QuESTEnv multiple times. Ignoring…

you know, even if it just says Ignoring... at the end, it's kind of weird to see such a message as an error because after that I thought that initialization of pyQuEST must be done only once! but how can it be done if this script will run multiple times on the different nodes? Or may be something might become in inconsistent state in case of multiple initialization?

Another case was related to the question that was came in mind how calculations was really distributed across nodes? until I found this hidden 💎 here https://github.com/QuEST-Kit/QuEST/blob/master/QuEST/include/QuEST.h#L504. Definitely, code well documented and supported with explanations but spread across different projects and files which adds some complexity in understanding.