Qiskit / benchpress

Apache License 2.0
19 stars 13 forks source link

Add a test of a 100q Clifford for device tranpilation #48

Closed ShellyGarion closed 1 month ago

ShellyGarion commented 1 month ago

@mtreinish suggested to add a random clifford example for a device transpliation. This circuit represent a random quantum circuit on the one hand, but also can be simulated effectively and verified using a classical computer on the other hand, thus providing a good example.

Here are the results for qiskit_gym (for a 100q random clifford):

                "circuit_operations": {
                    "sx": 126977,
                    "cz": 65542,
                    "rz": 14996,
                    "x": 398
                },
                "gate_count_2q": 65542,
                "depth_2q": 20004
            },
--------------------------------------- benchmark 'Transpile - Device': 1 tests ----------------------------------------
Name (time in s)                    Min      Max     Mean  StdDev   Median     IQR  Outliers     OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------
test_clifford_100_transpile     31.1481  31.1481  31.1481  0.0000  31.1481  0.0000       0;0  0.0321       1           1
------------------------------------------------------------------------------------------------------------------------
ShellyGarion commented 1 month ago

qiskit benchmark took me a few minutes, but tket and bqskit took more than 1 hour, and didn't finish, so I have no results for them yet

ShellyGarion commented 1 month ago

bqskit results (several hours):

                "circuit_operations": {
                    "RZGate": 411018,
                    "SqrtXGate": 274012,
                    "CZGate": 68453
                },
                "gate_count_2q": 68453,
                "depth_2q": 23920
            },
--------------------------------------------- benchmark 'Transpile - Device': 1 tests ----------------------------------------------
Name (time in s)                       Min         Max        Mean  StdDev      Median     IQR  Outliers     OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------
test_clifford_100_transpile     4,371.6338  4,371.6338  4,371.6338  0.0000  4,371.6338  0.0000       0;0  0.0002       1           1
------------------------------------------------------------------------------------------------------------------------------------
nonhermitian commented 1 month ago

So the test time looks to be something like an hour an a half, but I think you were saying it took something like 12 hours. I wonder if the QASM load time is an issue here?

ShellyGarion commented 1 month ago

So the test time looks to be something like an hour an a half, but I think you were saying it took something like 12 hours. I wonder if the QASM load time is an issue here?

I'm not sure how much time it took exactly, but at least several hours. So perhaps the QASM load was an issue. Also, tket tests crashed after several hours.

nonhermitian commented 1 month ago

I think it probably makes sense to write functions that just build the circuits directly in each SDK in this case. We could then add that build time to the creation tests as well provided the method was uniform across SDKs

ShellyGarion commented 1 month ago

Do you have examples of generating pseudo-random circuits? I see that for QV you use QASM files as well

nonhermitian commented 1 month ago

See here for example: https://github.com/Qiskit/benchpress/blob/46e4d6f5a5be4e2cf5c0ebc35facb9e4ac6afc22/benchpress/tket_gym/circuits/circuits.py#L20

there is a corresponding example in the other SDKs, except Staq

ShellyGarion commented 1 month ago

it seems that building circuits is done in construct/test_build.py::TestWorkoutCircuitConstruction while in this PR I added the tests in device_transpile/test_summit.py::TestWorkoutDeviceTranspile100Q, in order to benchmark the transpilation process. Should we perhaps merge this PR and I'll open another PR for circuit building ?

nonhermitian commented 1 month ago

I would just refactor this one since we would want the same functionality across the board.

ShellyGarion commented 1 month ago

Added 100q clifford circuit construction benchmarks.

Qiskit:

-------------------------------- benchmark 'Circuit construction': 1 tests ---------------------------------
Name (time in s)           Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------
test_clifford_build     3.6981  3.6981  3.6981  0.0000  3.6981  0.0000       0;0  0.2704       1           1
------------------------------------------------------------------------------------------------------------

Tket:

-------------------------------- benchmark 'Circuit construction': 1 tests ---------------------------------
Name (time in s)           Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------
test_clifford_build     2.0643  2.0643  2.0643  0.0000  2.0643  0.0000       0;0  0.4844       1           1
------------------------------------------------------------------------------------------------------------

BQSkit:

-------------------------------- benchmark 'Circuit construction': 1 tests ---------------------------------
Name (time in s)           Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------
test_clifford_build     4.8682  4.8682  4.8682  0.0000  4.8682  0.0000       0;0  0.2054       1           1
------------------------------------------------------------------------------------------------------------