We should periodically (e.g. once a day?) run a set of sample circuits to track performance of qFlex, both in terms of memory cost and simulation runtime. Some options for this:
Run each circuit that has a hand-crafted ordering, at a reasonable depth (e.g. runtime < 30s)
Pros: using a fixed circuit set gives clearer comparisons between runs
Cons: might be biased towards these specific circuits
Generate circuits and orderings for each grid (via generator.py and order_circuit_simulation.py) and run several for each (e.g. 30 each, runtime < 1s)
Pros: more varied coverage for different circuit layouts (but still restricted to QS circuits)
Cons: more difficult to compare between runs, still biased towards QS circuits
Create a tool for generating truly random circuits (i.e. no fixed placement for 2-qubit gates) and follow the plan in (2), using the new tool to generate circuits
Pros: best option for covering various circuit layouts
Cons: time cost of writing tool, potential for untested layouts to misbehave (e.g. excessive runtime or memory usage), very difficult to compare between runs
We should periodically (e.g. once a day?) run a set of sample circuits to track performance of qFlex, both in terms of memory cost and simulation runtime. Some options for this:
generator.pyandorder_circuit_simulation.py) and run several for each (e.g. 30 each, runtime < 1s)