BackendEstimator gives inconsistent results

LenaPer commented 1 year ago

Environment

Qiskit Terra version: 0.25.0
Python version: 3.11.3
Operating system: macOS ventura

What is happening?

When comparing BackendEstimator with the aer primitive on the same fake backend, I expect more or less the same results, but BackendEstimator gives really inconsistent results.

How can we reproduce the issue?

The code to reproduce and visualize the issue :

import numpy as np
from qiskit.circuit.library import TwoLocal
from qiskit.utils import algorithm_globals
from qiskit.quantum_info import SparsePauliOp
import matplotlib.pyplot as plt 
from qiskit.primitives import Estimator
from qiskit.primitives import BackendEstimator
from qiskit_aer.primitives import Estimator as AerEstimator
from qiskit_aer.noise import NoiseModel
from qiskit.providers.fake_provider import FakeNairobi, FakeGuadalupeV2, FakeGuadalupe
algorithm_globals.random_seed = 50
num_qubits = 3
ansatz = TwoLocal(num_qubits=num_qubits, rotation_blocks="ry", entanglement_blocks="cz")
op = SparsePauliOp.from_list([("III", 0.5),("IIZ", -0.5)])
estimator = Estimator()
ntest = 10
num_parameters = ansatz.num_parameters
parameters = np.random.rand(ntest, num_parameters)
#exact
job = estimator.run([ansatz]*ntest, [op]*ntest, parameters, shots = None)
exact_values = job.result().values

#aer noise
device = FakeGuadalupe()
seed = 170
coupling_map = device.configuration().coupling_map
noise_model = NoiseModel.from_backend(device)
basis_gates = device.configuration().basis_gates
estimator_aer_device = AerEstimator(
    backend_options={
        "coupling_map": coupling_map,
        "noise_model": noise_model,
        "basis_gates": basis_gates,
    },
    run_options={"seed": seed, "shots": 10000},
    transpile_options={"seed_transpiler": seed},
)

job = estimator_aer_device.run([ansatz]*ntest,[op]*ntest,parameters,shots = 10000)
aer_device_values = job.result().values
plt.scatter(exact_values,aer_device_values, label="AerEstimator with noise")

#backend estimator
device_estimator = BackendEstimator(device)
job = device_estimator.run([ansatz]*ntest,[op]*ntest,parameters,shots = 10000)
device_values = job.result().values
plt.scatter(exact_values,device_values, color="purple", label="BackendEstimator")
plt.legend(loc="upper left")
plt.plot([0,1],[0,1],'--', color="orange")

What should happen?

I expect the results to be similar to the aer one and consistent from one run to the other.

Any suggestions?

No response

priyansh-1902 commented 1 year ago

I tried to look into this

It seem that the BackendEstimator is working fine for fake_tokyo, fake_tenerife, fake_rueschlikon and fake_yorktown. So maybe the bug is specific to other fake_providers

fake_tokyo fake_tenerife fake_rueschlikon fake_yorktown