Fake backends ignore errors related to basis gates or number of qubits

[yaelbh]

Environment

• Qiskit Terra version: 0.20.0
• Python version: 3.9.5
• Operating system: Linux

What is happening?

Fake backends' behavior is sometimes different from their corresponding real devices.

Concretely, a device won't run a circuit that contains gates that do not belong to the set of basis gates. The fake backend will run the circuit.

Similarly, the device won't run a circuit that refers to non-existing qubits, while the fake backend will.

How can we reproduce the issue?

Code:

from qiskit import QuantumCircuit, IBMQ
from qiskit.providers.fake_provider import FakeLima
from qiskit.providers.backend import BackendV1

provider = IBMQ.load_account()
backends = [provider.backend.ibmq_lima, FakeLima()]

circ_not_a_basis_gate = QuantumCircuit(1, 1)
circ_not_a_basis_gate.h(0)
circ_not_a_basis_gate.measure(0, 0)
circ_not_a_basis_gate.name = "not-a-basis-gate"

circ_many_qubits = QuantumCircuit(10, 10)
circ_many_qubits.measure(9, 9)
circ_many_qubits.name = "many-qubits"

for circ in [circ_not_a_basis_gate, circ_many_qubits]:
    for backend in backends:
        print(f"\nRunning the {circ.name} circuit on {backend}:")

        job = backend.run(circ)

        try:
            print("Job result:", job.result())
        except Exception as e:
            print("Exception when trying to fetch job result:", e)

        print("Job status:", job.status())

Output:

Running the not-a-basis-gate circuit on ibmq_lima:
Exception when trying to fetch job result: "Unable to retrieve result for job 62f35351eace5881b6c259f4. Job has failed: The Qobj uses gates (['h']) that are not among the basis gates (['id', 'rz', 'sx', 'x', 'cx', 'reset']). Error code: 1106."
Job status: JobStatus.ERROR

Running the not-a-basis-gate circuit on fake_lima:
Job result: Result(backend_name='aer_simulator', backend_version='0.10.4', qobj_id='2b239bdf-2b05-49c9-9afe-e94e3175a530', job_id='2855837a-9efc-4659-8391-d2295e51e31a', success=True, results=[ExperimentResult(shots=1024, success=True, meas_level=2, data=ExperimentResultData(counts={'0x0': 525, '0x1': 499}), header=QobjExperimentHeader(clbit_labels=[['c', 0]], creg_sizes=[['c', 1]], global_phase=0.0, memory_slots=1, metadata={}, n_qubits=1, name='not-a-basis-gate', qreg_sizes=[['q', 1]], qubit_labels=[['q', 0]]), status=DONE, seed_simulator=3721132851, metadata={'parallel_state_update': 1, 'noise': 'superop', 'batched_shots_optimization': False, 'measure_sampling': True, 'device': 'CPU', 'num_qubits': 1, 'parallel_shots': 1, 'remapped_qubits': False, 'method': 'density_matrix', 'active_input_qubits': [0], 'num_clbits': 1, 'input_qubit_map': [[0, 0]], 'fusion': {'applied': False, 'max_fused_qubits': 5, 'enabled': True, 'threshold': 14}}, time_taken=0.004068663)], date=2022-08-10T09:42:30.224142, status=COMPLETED, header=QobjHeader(backend_name='aer_simulator', backend_version='0.10.4'), metadata={'time_taken': 0.019088631, 'time_taken_execute': 0.008525004, 'parallel_experiments': 1, 'omp_enabled': True, 'max_gpu_memory_mb': 0, 'num_mpi_processes': 1, 'time_taken_load_qobj': 0.010551124, 'max_memory_mb': 7068, 'mpi_rank': 0}, time_taken=0.01973700523376465)
Job status: JobStatus.DONE

Running the many-qubits circuit on ibmq_lima:
Exception when trying to fetch job result: 'Unable to retrieve result for job 62f353565838303af5f1df08. Job has failed: The number of qubits in the Qobj (10) is higher than the number of qubits supported by the device (5). Error code: 1109.'
Job status: JobStatus.ERROR

Running the many-qubits circuit on fake_lima:
Job result: Result(backend_name='aer_simulator', backend_version='0.10.4', qobj_id='bb0e3ece-bf05-4dae-bf46-5eff578199e0', job_id='8aa2036f-6d82-4c53-bf79-23a1d549fff5', success=True, results=[ExperimentResult(shots=1024, success=True, meas_level=2, data=ExperimentResultData(counts={'0x0': 1024}), header=QobjExperimentHeader(clbit_labels=[['c', 0], ['c', 1], ['c', 2], ['c', 3], ['c', 4], ['c', 5], ['c', 6], ['c', 7], ['c', 8], ['c', 9]], creg_sizes=[['c', 10]], global_phase=0.0, memory_slots=10, metadata={}, n_qubits=10, name='many-qubits', qreg_sizes=[['q', 10]], qubit_labels=[['q', 0], ['q', 1], ['q', 2], ['q', 3], ['q', 4], ['q', 5], ['q', 6], ['q', 7], ['q', 8], ['q', 9]]), status=DONE, seed_simulator=3017945355, metadata={'parallel_state_update': 1, 'noise': 'superop', 'batched_shots_optimization': False, 'measure_sampling': True, 'device': 'CPU', 'num_qubits': 1, 'parallel_shots': 1, 'remapped_qubits': True, 'method': 'density_matrix', 'active_input_qubits': [9], 'num_clbits': 10, 'input_qubit_map': [[9, 0]], 'fusion': {'applied': False, 'max_fused_qubits': 5, 'enabled': True, 'threshold': 14}}, time_taken=0.006464472)], date=2022-08-10T09:42:33.850333, status=COMPLETED, header=QobjHeader(backend_name='aer_simulator', backend_version='0.10.4'), metadata={'time_taken': 0.044194349, 'time_taken_execute': 0.013991019, 'parallel_experiments': 1, 'omp_enabled': True, 'max_gpu_memory_mb': 0, 'num_mpi_processes': 1, 'time_taken_load_qobj': 0.030191295, 'max_memory_mb': 7068, 'mpi_rank': 0}, time_taken=0.04497790336608887)
Job status: JobStatus.DONE

What should happen?

My understanding of the role of the fake backends is that they should mimic as much as possible the behavior of the device. If this view is correct, then in the code snippet above they should raise exceptions similar to the device, and set the job status to ERROR.

Any suggestions?

No response

[mtreinish]

As I stated in #9439 right now the fake backends pass the circuit directly to aer (or basicaer if aer isn't installed) with a noise model set to simulate the circuit with the backend's properties. If aer isn't enforcing this for us based on the noise model then we will have to do validation on input circuits in the fake backend's run method directly. Basically just check that all(x.num_qubits == self.num_qubits for x in run_input) and a similar check for basis gates on each input before passing the circuit to the simulator.

[Theo-Reignier]

Hi there, Can I be assigned to this issue please?

[NelsonOoi]

Hi, if this issue hasn't been assigned and has no WIP PRs at the moment, @andrea-miramonserr and I would like to work on this! Could we be assigned to the issue?

[AngeloDanducci]

@Theo-Reignier @NelsonOoi @Andrea-MiramonSerr Hi all, you've expressed interest in this issue previously but it slipped through the cracks. If you are still interested in and available to contribute to this issue please ping me here and I will assign the issue to the first responder.

[JustinWoodring]

@AngeloDanducci, I'd like to take this issue if that's okay! (I'm also a newcomer 😄)

[AngeloDanducci]

@JustinWoodring assigned, thanks!