search

qiboteam / qibojit-benchmarks

Benchmark code for qibojit performance accessment
Apache License 2.0
2 stars 3 forks source link

Different performances between main.py and compare.py #15

Closed mlazzarin closed 2 years ago

mlazzarin commented 2 years ago

If I do

python main.py --nqubits something --backend qibojit

and

python compare.py --nqubits something --library qibojit

I obtain different results (even a 10-25% difference in simulation time). Is this the expected behavior?

mlazzarin commented 2 years ago

I found out that the reason of this discrepancy is that compare.py converts the final state vector to numpy by default, while main.py does not. If I use

python main.py --nqubits something --backend qibojit --transfer

and I consider also the transfer times, the results are consistent with those of compare.py.

stavros11 commented 2 years ago

I confirm that the difference is due to the transfer time, particularly when a GPU is used. Here are some numbers for 28 qubits:

circuit main.py simulation main.py transfer compare.py simulation
qft 1.69758 1.44701 2.96322
variational 0.72737 1.39814 2.13173
bv 0.81455 1.20676 2.02743
supremacy 0.91002 1.38992 2.31580
qv 1.51656 1.39442 2.75839

The reason I used the --transfer flag in main.py is that some time in the past we were interested in logging the CPU-GPU transfer time seperately. It is not possible to have the same distinction in compare.py because most libraries other than qibo return the final state vector as numpy array so re-casting to numpy makes no sense.

If we would like it to be consistent we can remove the transfer from main.py and do it automatically. Otherwise we can leave it as it is and keep in mind this difference between the two scripts. Note that when using compare.py the transfer is done automatically during execution (see here).

scarrazza commented 2 years ago

Another problem related to not using --transfer is that tensorflow may not evaluate the final state if this object is not used anywhere else, so transfer forces the execution and represents the end-point of the computation.

mlazzarin commented 2 years ago

Ok, thanks for the replies. I think we may simply leave main.py as it is and remember that the time is logged differently between the two scripts. The transfer time may still be useful for profiling qibo. What do you think?

We should just be sure that we don't use main.py to compare qibo with other libraries.