More efficient job management

[nkanazawa1989]

What is the expected behavior?

Recently I did tons of pulse calibration experiments and this is the feedback from my experience. I understand this requires kind of metadata and this cannot be immediately implemented. So please consider this as one of feedback from usability testing.

Definitions:

• I refer to Schedule and QuantumCircuit as program
• I refer to a set of information that identifies a set of program and its run configuration as entry
• I refer to a combination of programs and entry as experiment, i.e. single VQE loop or calibration protocol

1. More efficient job execution.

In current IBMQJobManager we can set only max_experiments_per_job and the provider only split programs into several jobs. However, we also want to merge several experiments into single job to save queuing time. So provider should be able to sort experiments by run configuration and then pack experiments into a single job if they have the same run configuration.

(example) When we calibrate a quantum chip, we may split qubits into several sub groups to avoid spectator interaction. To execute pulse power calibration, I create calibration programs for each qubit group and generate several experiment objects. Those experiments can be simultaneously submited to the backend.

qubit_group1 = [0, 2, 4]
qubit_group2 = [1, 3]

exp1 = amp_calibration(qubit_group1, shots=1024, **parameters) # -> new experiment
exp2 = amp_calibration(qubit_group2, shots=1024, **parameters) # -> new experiment

run(exp1, exp2) # -> submit single job

But once total number of experiment exceeds the limit, the provider needs to submit multiple jobs.

run(exp1, exp2, exp3, exp4, exp5) # -> submit two jobs

Or the provider needs to submit multiple jobs if the run configuration is different.

exp1 = amp_calibration(qubit_group1, shots=1024, **parameters) # -> new experiment
exp2 = amp_calibration(qubit_group2, shots=2048, **parameters) # -> new experiment

run(exp1, exp2) # -> submit two jobs

Basically I don't want to manage job submission at this level.

2. More reusable experimental results

The iterative type experiments such as VQE and closed-loop calibration tend to generate huge amount of experimental data. The IBMQJobManager can manage submission of a large set of programs, but it just merges the independent job result behind the scene (no unique identifier for the set of jobs).

In general experiment procedures (of course this depends on your preference), we rarely store the raw experimental result object or analyzed result, i.e. image file with fit curve, but we just keep a log file that contains final outcome of each iteration loop. After the experiment is completed, we check the log file, retrieve the result and analyze it manually only when we find something unexpected.

If I submit a huge experiment, i.e. 3 qubit process tomography, to IBMQJobManager it will submit two or three jobs with independent job ids and just merge the retrieved results. This is what I did in the paper of "Experimental implementation of non-Clifford interleaved randomized benchmarking with a controlled-S gate", however this resulted in overhead of managing hundreds bytes of result objects and I gave up using github for project management.

If I want to retrieve the specific results without locally storing the returned object, it is hard to recover a set of job ids that consist of the experiment. Recently I started managing the experiment with this dataclass and it works well so far.

@dataclass
class DatabaseEntry:
    """Python dataclass to represent single experimental routine with parameter sweep (xvals).

    This object is mutable and should be provided for each experiment."""
    experiment: str
    qubits: List[int]
    xvals: List[float]
    parameters: Optional[Dict[str, Union[int, float, str]]] = None
    job_id: Optional[str] = None
    position: Optional[int] = None
    length: Optional[int] = None
    run_date: Optional[datetime] = None
    shots: int = None
    meas_level: int = 2
    meas_return: str = None
    success: bool = False

• Note that in this model I don't assume executing any huge experiments. so all programs can be assigned to single job id.
• However single job id can be associated with multiple experiment entry (n:1 mapping).
• It is also possible to create new model that single experiment entry consists of multiple job ids (1:n mapping).

I usually search the database (of course this is local one) for specific entry by filtering with keyword and some conditions that I can find in the experiment log file:

entries = search_entry('my_database', experiments='amp_calibration', date_from=t1, date_to=t2)
format_entries(entries)

This outputs the table like:

===================================================================================================================
 index| experiment         | qubits    | parameters                              | run date                      |
-------------------------------------------------------------------------------------------------------------------
 0    | amp_calibration    | 0,2,4     | no value                                | 2020-09-06T08:09:45+00:00     |
 1    | amp_calibration    | 1,3       | no value                                | 2020-09-06T08:09:45+00:00     |
-------------------------------------------------------------------------------------------------------------------

Then I can retrieve a set of results for the specific experiment entry:

result = get_result(backend, entries[1])

I believe this kind of stuff had been already discussed, but if you need the detailed feedback of my experience please let me know.

[blakejohnson]

FYI, the word "program" is going to mean something else with the new Program module and runtime. So, you may want to look for a different word.

[blakejohnson]

Also, the experiment database functionality is coming soon to the ibmq provider: https://github.com/jyu00/qiskit-ibmq-provider/blob/result_db/qiskit/providers/ibmq/experiment/experiment.py

However, having a local file store that implements the same API as the provider would be extremely nice.

[jyu00]

@nkanazawa1989 I think I understand your first point pretty well, some questions about the second point:

If I want to retrieve the specific results without locally storing the returned object, it is hard to recover a set of job ids that consist of the experiment.

When you invoke job_manager.run(), a ManagedJobSet object is returned and has a unique ID. You can use this job set id to retrieve the jobs associated with the run. You're right, however, that there is no easy way to retrieve only the results of a specific job without downloading other job results in the set.

So I think with the second point you're looking for:

• Be able to download the minimum amount of data necessary to get the results for a single experiment or a circuit in an experiment (e.g. experiment.get_counts(5))
• Better filters for job set querying.

Does this sound right?

[kt474]

Stale issue, closing