Pulse benchmarks

[taalexander]

What is the expected behavior?

The circuit, transpilation and circuit assembly modules are well benchmarked. There should be associated benchmarks for the pulse path and an associated integration benchmark from the construction of a relatively sophisticated quantum circuit, default backend transpilation and scheduling.

Benchmarks:

• [ ] Schedule construction
  • with parameters being the:
    • [x] number of unique pulses
    • [x] number of channels
    • [x] ~number of uses of each pulse~
  • options:
    • pulse types:
      • [x] Parametric
      • [x] Sample
    • Building methods:
      • [x] Append instruction
      • [x] Insert instruction right to left
      • [x] Convert the instruction to a schedule and then take the union of all schedules.
• [ ] Scheduling. Use benchmarks from transpiler but decomposed to a native single qubit gate. Use mockbackend of 20q device:
  • for schedules:
    • [ ] single gate schedule
    • [ ] cx gate schedule
    • [ ] measure schedule
    • [ ] large qasm schedule
  • scheduling methods:
    • [ ] alap
    • [ ] asap
• [ ] End to end benchmark. Add pulse benchmarks to:
  • [ ] quantum volume
  • [ ] randomized benchmarking
• [ ] Assembly
  • [ ] random schedule tests

[nkanazawa1989]

how about adding error mitigation as a potential target? it requires stretching of pulses and may have a different aspect from others.

[taalexander]

We don't currently have an error mitigation module so this would be infeasible to add as a benchmark. All of the benchmarks above already have code for circuit generation.

[nkanazawa1989]

Yes. This is why I said potential, just for future plan. I agree the list you have is enough to cover many practical situations.

[taalexander]

Discussing with @kdk and @mtreinish it appears the current benchmarks are already taking a significant fraction of a day to run, we may wish to pare down the list. Any suggestions for what to cut @nkanazawa1989?

[nkanazawa1989]

In that sense, benchmarks that has less dependency on circuit transpiler, ie benchmarks that don't intend to evaluate connectivity, are good candidate. There are two reasons:

• overlap with circuit benchmarks
• unwanted performance dependency

For example, QV and random hex may insert many SWAP operations, depending on topology, and end to end performance depends on the result of qubit routing and basis gates (if swap exists) rather than scheduling algorithm. If the purpose of pulse benchmark is to evaluate the performance of scheduling algorithm, I think 2q RB is good candidate.

[taalexander]

I think it is still important to have some end to end benchmarks that have interdependency. This is valuable to show realistic performance numbers. QV is a great example I think.

[taalexander]

@mtreinish, I don't think this should have been closed as this is an epic with follow up issues coming.

[SooluThomas]

It automatically got closed because, in that PR, I added the summary as "partially fixes" this issue.

[jakelishman]

@nkanazawa1989: I've moved this from the metapackage to Terra, since after #10546 it'll be here. Could you see if this is still something you'll be wanting to do, or close the issue if not?