IBMRuntimeError - Max retries exceeded with url

[maniraman-periyasamy]

Describe the bug

The following error is thrown when the qiskit runtime estimator.run() function is called sequentially inside a loop.

'Failed to run program: "HTTPSConnectionPool(host=\'runtime.de.quantum-computing.ibm.com\', port=443): Max retries exceeded with url: /jobs (Caused by ResponseError(\'too many 500 error responses\'))"'

The first call works as expected, however the estimator.run() function throws the error from the second call on.

Steps to reproduce

observables = [SparsePauliOp(Pauli("ZZII")), SparsePauliOp(Pauli("IIZZ"))]
options = Options()
options.resilience_level = 2
options.optimization_level = 0
EPOCHS = 20
epoch = 0
with Session(service=service, backend="ibmq_ehningen") as session:
    executer = Estimator(session=session, options=options)
    while epoch<EPOCHS:
        job = executer.run(circuits=[circuit]*2, shots=1024, observables=observables)
        #job = wait_for_termination(job)
        exp = job.result().values
        print(f"Epoch {epoch}: {exp}")
        epoch +=1

Expected behavior

Estimator outputs expectation value sequentially inside the loop.

Suggested solutions

Additional Information

This bug wasn't there during Dec-2022 (Last I accessed ibmq_enhingen device)

• qiskit-ibm-runtime version: 0.8.0
• Python version: 3.10.8
• Operating system: ubuntu 20.04

[kt474]

@maniraman-periyasamy can you give an example circuit where this happens?

[maniraman-periyasamy]

@kt474 I have attached the minimal example script (as txt file) which reproduces that error when accessing ibmq_ehningen device.

runtime_error_minimal_example.txt

[kt474]

There is a bug related to session times on the server side causing these timeout errors - we are working on a fix

[adekusar-drl]

I experience the same problem!

[luciacuervovalor]

I am experiencing the same problem as well. I am using a retry sampler (https://github.com/Qiskit/qiskit-ibm-runtime/issues/598#issuecomment-1431397974) that, so far, had allowed me to avoid most of the connection errors I had come across. However, since Tuesday, I get consistently a "too many 500 error responses" error after every retry (so, at every iteration). I have tried setting the max_time myself to 28,500 seconds when initialising the Session, to avoid it being accidentally set to <300s, but this hasn't helped either.

[kt474]

@maniraman-periyasamy @adekusar-drl @luciacuervovalor we just deployed an update on the server side - can you rerun your jobs?

[maniraman-periyasamy]

@kt474 Yup. It works now!