Closed ShellyGarion closed 1 month ago
qiskit benchmark took me a few minutes, but tket and bqskit took more than 1 hour, and didn't finish, so I have no results for them yet
bqskit results (several hours):
"circuit_operations": {
"RZGate": 411018,
"SqrtXGate": 274012,
"CZGate": 68453
},
"gate_count_2q": 68453,
"depth_2q": 23920
},
--------------------------------------------- benchmark 'Transpile - Device': 1 tests ----------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------
test_clifford_100_transpile 4,371.6338 4,371.6338 4,371.6338 0.0000 4,371.6338 0.0000 0;0 0.0002 1 1
------------------------------------------------------------------------------------------------------------------------------------
So the test time looks to be something like an hour an a half, but I think you were saying it took something like 12 hours. I wonder if the QASM load time is an issue here?
So the test time looks to be something like an hour an a half, but I think you were saying it took something like 12 hours. I wonder if the QASM load time is an issue here?
I'm not sure how much time it took exactly, but at least several hours. So perhaps the QASM load was an issue. Also, tket tests crashed after several hours.
I think it probably makes sense to write functions that just build the circuits directly in each SDK in this case. We could then add that build time to the creation tests as well provided the method was uniform across SDKs
Do you have examples of generating pseudo-random circuits? I see that for QV you use QASM files as well
See here for example: https://github.com/Qiskit/benchpress/blob/46e4d6f5a5be4e2cf5c0ebc35facb9e4ac6afc22/benchpress/tket_gym/circuits/circuits.py#L20
there is a corresponding example in the other SDKs, except Staq
it seems that building circuits is done in construct/test_build.py::TestWorkoutCircuitConstruction
while in this PR I added the tests in device_transpile/test_summit.py::TestWorkoutDeviceTranspile100Q
, in order to benchmark the transpilation process.
Should we perhaps merge this PR and I'll open another PR for circuit building ?
I would just refactor this one since we would want the same functionality across the board.
Added 100q clifford circuit construction benchmarks.
Qiskit:
-------------------------------- benchmark 'Circuit construction': 1 tests ---------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------
test_clifford_build 3.6981 3.6981 3.6981 0.0000 3.6981 0.0000 0;0 0.2704 1 1
------------------------------------------------------------------------------------------------------------
Tket:
-------------------------------- benchmark 'Circuit construction': 1 tests ---------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------
test_clifford_build 2.0643 2.0643 2.0643 0.0000 2.0643 0.0000 0;0 0.4844 1 1
------------------------------------------------------------------------------------------------------------
BQSkit:
-------------------------------- benchmark 'Circuit construction': 1 tests ---------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------
test_clifford_build 4.8682 4.8682 4.8682 0.0000 4.8682 0.0000 0;0 0.2054 1 1
------------------------------------------------------------------------------------------------------------
@mtreinish suggested to add a random clifford example for a device transpliation. This circuit represent a random quantum circuit on the one hand, but also can be simulated effectively and verified using a classical computer on the other hand, thus providing a good example.
Here are the results for qiskit_gym (for a 100q random clifford):