RB performance benchmarks

Qiskit / qiskit-metapackage

Qiskit is an open-source SDK for working with quantum computers at the level of circuits, algorithms, and application modules.

https://qiskit.org

Apache License 2.0

3.03k stars 751 forks source link

RB performance benchmarks #272

Closed ajavadia closed 4 years ago

ajavadia commented 5 years ago

What is the expected behavior?

Let's add randomized benchmarking from Ignis as one of the benchmarks under test/benchmarks. It is helpful for tracking performance.

dongreenberg commented 5 years ago

How do you feel about adding some applications benchmarks (e.g. VQE, operator estimation, hamiltonian simulation)? I can start a separate bug to discuss if you think that could be valuable.

mtreinish commented 5 years ago

@dongreenberg I definitely think that having some application benchmarks would be useful. In the first WIP version of the benchmark suite I had a vqe benchmark (based on terra's optimizers module which were removed in 0.7) but removed it by the time things finalized because the terra optimization module it depended on was removed.

ShellyGarion commented 5 years ago

@ajavadia - I started a PR with RB performance benchmark.

How can I run the performance benchmark?
Is there some link to the performance results and graphs?
Which RB configurations would you suggest to test? I started with 2-qubits RB, with 50 different Clifford lengths (going from 1 Clifford to 200 Cliffords, spaced linearly) and 20 seeds.

mtreinish commented 5 years ago

@ShellyGarion You run the benchmarks using the asv. I normally run things locally for testing with just asv dev but you can also use asv run especially if you want to test across the history. You can see current benchmark results here: http://169.61.88.206/

ajavadia commented 5 years ago

@ShellyGarion I'm not sure varying the seed affects things much from a performance point of view (you can try). The Clifford lengths are good. Maybe you can add 1Q RB as well.

@dongreenberg yes please add a couple applications too. Maybe one with a huge circuit, and one with many small circuits.

ShellyGarion commented 5 years ago

@mtreinish - the link for current benchmark results http://169.61.88.206/ does not work for me anymore

mtreinish commented 5 years ago

Yes, I implemented a more permanent hosting solution for the benchmark results since I left that comment last month. You can view the benchmark results here: https://qiskit.github.io/qiskit/

ShellyGarion commented 5 years ago

@ajavadia - I added several more parameters:

1Q, 2Q, and simultaneous 1Q&2Q.
1, 5, 10 seeds (need to check if no. of seeds affects the performance).
Clifford lengths: 1 to 200 in steps of 4, 1 to 500 in steps of 10.

ShellyGarion commented 5 years ago

The RB performance benchmark is now reduced to the following parameters:

1Q, 2Q, and 3Q simultaneously 1Q&2Q, namely: rb_pattern=[[[0]], [[0, 1]], [[0, 2], [1]]]

with the following default parameters:

1 seed
Clifford lengths: 1 to 200 in steps of 4, namely: length_vector = np.arange(1, 200, 4)

ShellyGarion commented 4 years ago

I think that this old issue could be closed @ajavadia @mtreinish Here is a link to RB performance test: https://github.com/Qiskit/qiskit/blob/master/test/benchmarks/randomized_benchmarking.py

mtreinish commented 4 years ago

Yep, this has been done since #306 and can be closed.