Add multi-GPU option for qibo library

[stavros11]

@mlazzarin I realized that I had an old implementation of multigpu benchmarks for qibo in this repo and I updated using the latest libraries branch to avoid doing double work. The multigpu configuration can be passed in --library-options using the existing benchmark scripts, for example:

python compare.py --library qibo --nqubits 31 --library-options accelerators=1/GPU:0+1/GPU:1

Let me know if you agree. Note that you can "reuse" a single GPU by passing accelerators=2/GPU:0 (works for >2 times too, but should be power of 2).

I checked running the above for a few configurations and it seems to execute for both qibojit and qibotf. I am not sure if the results make sense, though. I suspect that parallelization when multiple GPUs are used is broken for qibojit, because for qibotf I get 100% simultaneously on all devices on nvidia-smi, while for qibojit it seems to run sequentially. I will investigate this further. We also need to add tests that check the final state for multi-gpu configurations here.

[stavros11]

Here are some results on the DGX in double precision:

QFT

| nqubits (accelerators) | qibojit dry run (sec) | qibojit simulation (sec) | qibotf dry run (sec) | qibottf simulation (sec) | |:-------------------------------------|-----------------------:|--------------------------------:|----------------------:|-------------------------------:| | 31 (2/GPU:0) | 75.1929 | 68.7758 | 82.987 | 83.3789 | | 31 (1/GPU:0 + 1/GPU:1) | 67.636 | OOM | 55.9763 | 55.613 | | 32 (4/GPU:0) | 195.4 | 188.271 | 219.633 | 220.952 | | 32 (2/GPU:0 + 2/GPU:1) | 155.852 | 137.265 | 137.71 | 138.196 | | 32 (1/GPU:0 + 1/GPU:1 + 1/GPU:2 + 1/GPU:3) | 135.925 | OOM | 96.0887 | 94.1834 |

Variational

| nqubits (accelerators) | qibojit dry run (sec) | qibojit simulation (sec) | qibotf dry run (sec) | qibottf simulation (sec) | |:-------------------------------------|-----------------------:|--------------------------------:|----------------------:|-------------------------------:| | 31 (2/GPU:0) | 61.958 | 55.7334 | 74.2878 | 75.3073 | | 31 (1/GPU:0 + 1/GPU:1) | 59.6055 | 38.3918 | 50.0096 | 50.3635 | | 32 (4/GPU:0) | 167.434 | 160.963 | 217.574 | 217.58 | | 32 (2/GPU:0 + 2/GPU:1) | 127.991 | 109.543 | 146.671 | 148.087 | | 32 (1/GPU:0 + 1/GPU:1 + 1/GPU:2 + 1/GPU:3) | 121.044 | 75.4436 | 111.084 | 110.45 |

The issues to be resolved from the qibo side are the following:

• [ ] Why the second run in QFT runs out of memory when many devices are used, while the same run works for qibotf. I suspect that some objects are not deleted properly but it is strange that the same issue does not appear in the variational circuit.
• [ ] It seems that some kind compilation is happening in qibojit dry run as in all cases it is much slower than simulation, in contrast to qibotf where both runs are similar. I also believe that the joblib parallelization does not work very well with qibojit.

Despite these issues, it seems that qibojit generally gets better simulation times than qibotf, most likely due to faster CPU-GPU communication.

[stavros11]

In order to demonstrate the second issue with parallelization, here are some plots that show the GPU utilization as captured from nvidia-smi every short time interval (0.05 sec). For qibotf we see that the two GPUs are working simultaneously, while for qibojit most operations seem to be applied sequentially.

qibojit - QFT - 31 qubits


qibotf - QFT - 31 qubits


qibojit - Variational - 31 qubits


qibotf - Variational - 31 qubits


[scarrazza]

I suppose the joblib configuration is the same between both backends, right? If that is the case, then maybe there is some extra cuda sync which is blocking the operation in qibojit.

[stavros11]

I suppose the joblib configuration is the same between both backends, right?

Yes, the multigpu circuit is defined in qibo and is the same for both backends. This is the only place where joblib is used.

If that is the case, then maybe there is some extra cuda sync which is blocking the operation in qibojit.

If you mean the cp.cuda.stream.get_current_stream().synchronize() that we do in cupy backends, I tried removing it from everywhere (all qibo and qibojit tests still pass without the sync), but the multigpu situation remains the same. Also the OOM issue in the second QFT run remains.

[stavros11]

I explored the memory issue a bit more and found the following:

1. It happens with smaller qubit numbers. In this case there's no OOM but additional GPU memory is occupied. For example for 28 qubits distributed to two physical GPUs, the dry run occupies 2GB in both GPUs, while the second run occupies 4GB in the first GPU and 2GB in the second. It seems that the state is not cleaned properly in the first GPU after the dry run.
2. If I do three or more repetitions (dry run + two more reps) for 28 qubits, memory occupation remains 4GB + 2GB during all reps. So the non-cleaning issue only happens in the dry run, additional reps do not increase memory.
3. The issue appears for the QFT circuit and only when the definition used in the current repo is used. For example, if we do

   from qibo.models import QFT
   c = models.QFT(31, accelerators={"/GPU:0": 1, "/GPU:1": 1})
   final_state = c().numpy()  # dry run
   final_state = c().numpy()  # second run

   the OOM does not appear. More explicitly, if we define the QFT as:

   circuit = Circuit(nqubits)
   for i1 in range(nqubits):
   circuit.add(gates.H(i1))
   for i2 in range(i1 + 1, nqubits):
       theta = math.pi / 2 ** (i2 - i1)
      circuit.add(gates.CU1(i2, i1, theta))
   for i in range(nqubits // 2):
   circuit.add(gates.SWAP(i, nqubits - i - 1))

   then the OOM / double memory issue appears, while if we define it using the _DistributedQFT method from qibo the OOM does not appear. Note that the two circuits are equivalent, just some commuting gates are reordered and I checked that the final states of both are the same, even with random initial state. I am not sure if the issue appears with other circuits too.

[scarrazza]

Interesting, so the state vector is not being cleaned between runs. What happens if you force the state vector delete between the dry-run and execution using the code from this repo?

[stavros11]

Interesting, so the state vector is not being cleaned between runs. What happens if you force the state vector delete between the dry-run and execution using the code from this repo?

All the issues I wrote above are the same regardless of whether I delete the result (the execution output) after each run. I would assume that there is a bug and in the second circuit, which creates the problem, a reference to the state remains somewhere and that's why is not properly cleaned but this does not explain why the problem does not appear with qibotf. Also the problem remains even if I delete both the result and the whole circuit after each run.

[scarrazza]

Did you try with cp._default_memory_pool.free_all_blocks()?

[stavros11]

Did you try with cp._default_memory_pool.free_all_blocks()?

Yes, I also tried that one after object deletion and it does not make a difference.

[stavros11]

Here are some benchmarks with the latest version of qibojit, after merging the memory duplication fix.

qft

| nqubits (accelerators) | dry_run_time_qibojit | simulation_times_mean_qibojit | dry_run_time_qibotf | simulation_times_mean_qibotf | |:-------------------------------------|-----------------------:|--------------------------------:|----------------------:|-------------------------------:| | 31 (2/GPU:3) | 75.1272 | 68.4773 | 84.8177 | 85.2387 | | 31 (1/GPU:2+1/GPU:3) | 56.0591 | 46.028 | 56.27 | 55.7507 | | 32 (4/GPU:3) | 199.658 | 190.287 | 220.224 | 220.174 | | 32 (2/GPU:2+2/GPU:3) | 144.643 | 129.706 | 138.685 | 141.355 | | 32 (1/GPU:0+1/GPU:1+1/GPU:2+1/GPU:3) | 125.408 | 98.043 | 99.0873 | 96.8065 |

variational

| nqubits (accelerators) | dry_run_time_qibojit | simulation_times_mean_qibojit | dry_run_time_qibotf | simulation_times_mean_qibotf | |:-------------------------------------|-----------------------:|--------------------------------:|----------------------:|-------------------------------:| | 31 (2/GPU:3) | 61.8927 | 56.3496 | 75.4588 | 76.1184 | | 31 (1/GPU:2+1/GPU:3) | 48.8197 | 44.9677 | 51.4959 | 51.3772 | | 32 (4/GPU:3) | 177.81 | 171.728 | 227.636 | 230.986 | | 32 (2/GPU:2+2/GPU:3) | 140.191 | 134.599 | 157.079 | 159.028 | | 32 (1/GPU:0+1/GPU:1+1/GPU:2+1/GPU:3) | 125.215 | 90.3429 | 120.335 | 118.85 |

supremacy

| nqubits (accelerators) | dry_run_time_qibojit | simulation_times_mean_qibojit | dry_run_time_qibotf | simulation_times_mean_qibotf | |:-------------------------------------|-----------------------:|--------------------------------:|----------------------:|-------------------------------:| | 31 (2/GPU:3) | 64.1213 | 57.8321 | 76.3821 | 76.6204 | | 31 (1/GPU:2+1/GPU:3) | 49.0247 | 38.3491 | 51.978 | 51.9739 | | 32 (4/GPU:3) | 179.127 | 173.83 | 232.332 | 233.252 | | 32 (2/GPU:2+2/GPU:3) | 131.618 | 120.149 | 157.172 | 158.845 | | 32 (1/GPU:0+1/GPU:1+1/GPU:2+1/GPU:3) | 120.386 | 92.9109 | 118.797 | 120.125 |

Note that some other things are running in parallel in the machine, so there may be some noise and we cannot do a fair comparison with the tables above. I ran qibojit and qibotf sequentially though so that at least we can compare the two.

There appears to be an issue with the 4x GPU run of qibojit, I will rerun this to confirm whether it is something code related or just something temporary with the machine. Other than that qibojit times, even for dry run, are competitive with qibotf.

[scarrazza]

Thanks @stavros11, performance is quite good, despite the unsync initial step.

[stavros11]

@scarrazza, I updated the post above with the latest results for three different circuits. I believe qibojit performance is acceptable compared to qibotf, even for dry run. Perhaps it is possible to improve with a better multigpu approach but it may be interesting to compare with cuquantum first, after they release.

[scarrazza]

Looks good to me, can we merge?

[stavros11]

Looks good to me, can we merge?

Yes, this is okay from my side.