search

jpmorganchase / QOKit

QOKit
https://www.jpmorgan.com/technology/applied-research
Apache License 2.0
51 stars 23 forks source link

Communication inside the MPI simulator may cause overflow error when the message is large #42

Open HaoTy opened 8 months ago

HaoTy commented 8 months ago

This happens, for example, when calling MPI simulator's get_probabilities() which does self._comm.allgather(result.copy_to_host()). If the vector is too large (e.g. 32 qubits), the following is raised:

  File "mpi4py/MPI/Comm.pyx", line 1595, in mpi4py.MPI.Comm.allgather
  File "mpi4py/MPI/msgpickle.pxi", line 862, in mpi4py.MPI.PyMPI_allgather
  File "mpi4py/MPI/msgpickle.pxi", line 147, in mpi4py.MPI.pickle_dump
  File "mpi4py/MPI/msgbuffer.pxi", line 50, in mpi4py.MPI.downcast
OverflowError: integer 34359738509 does not fit in 'int'

which is due to a legacy issue of mpi4py/pickle (see https://github.com/mpi4py/mpi4py/issues/119) that is solved by mpi4py.util.pkl5. It seems the switch to pkl5 can be realized by simply changing the current self._comm = MPI.COMM_WORLD to pkl5.Intracomm(MPI.COMM_WORLD).

rsln-s commented 8 months ago

@danlkv, would you be able to take a look at this?