search

Qiskit / qiskit-addon-cutting

Reduce width and depth of quantum circuits by cutting gates and wires.
https://qiskit.github.io/qiskit-addon-cutting/
Apache License 2.0
80 stars 28 forks source link

Timeout during `generate_cutting_experiments` in `how_to_generate_exact_sampling_coefficients.ipynb` #515

Closed garrison closed 6 months ago

garrison commented 7 months ago

Now that #514 is merged, we can actually see which notebook hangs intermittently on CI.

=================================== FAILURES ===================================
_ /home/runner/work/circuit-knitting-toolbox/circuit-knitting-toolbox/docs/circuit_cutting/how-tos/how_to_generate_exact_sampling_coefficients.ipynb _
A cell timed out while it was being executed, after 300 seconds.
The message was: Cell execution timed out.
Here is a preview of the cell contents:
-------------------
subexperiments, coefficients = generate_cutting_experiments(
    circuits=subcircuits,
    observables=subobservables,
    num_samples=np.inf,
)
coefficients
-------------------

------------------------------ Captured log call -------------------------------
ERROR    traitlets:client.py:845 Timeout waiting for execute reply (300s).

Full log at:

garrison commented 6 months ago

I keep on adding to the list above, but so far this only seems to happen on Python 3.8 on Ubuntu.

garrison commented 6 months ago

I noticed a warning when running the test suite under Python 3.12 that may be a clue.

test/cutting/test_cutting_workflows.py: 48 warnings
test/cutting/test_cutting_roundtrip.py: 420 warnings
  /usr/local/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=17907) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

I was initially confused because we are not using multiprocessing explicitly, but it turns out that Qiskit's transpiler actually uses multiprocessing when a pass manager is asked to transpile multiple circuits in the same call. Indeed, the traceback of the warning is

circuit_knitting/cutting/cutting_experiments.py:188: in generate_cutting_experiments
    def test_generate_cutting_experiments(self):
        with self.subTest("simple circuit and observable"):
            qc = QuantumCircuit(2)
            qc.append(
                TwoQubitQPDGate(QPDBasis.from_instruction(CXGate()), label="cut_cx"),
                qargs=[0, 1],
            )
            comp_coeffs = [
                (0.5, WeightType.EXACT),
                (0.5, WeightType.EXACT),
                (0.5, WeightType.EXACT),
                (-0.5, WeightType.EXACT),
                (0.5, WeightType.EXACT),
                (-0.5, WeightType.EXACT),
            ]
>           subexperiments, coeffs = generate_cutting_experiments(
                qc, PauliList(["ZZ"]), np.inf
            )

test/cutting/test_cutting_experiments.py:52: 
    subexperiments_dict[label] = pass_manager.run(subexperiments)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:420: in wrapper
circuit_knitting/cutting/cutting_experiments.py:188: in generate_cutting_experiments
    return meth(*meth_args, **meth_kwargs)
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:182: in run
    subexperiments_dict[label] = pass_manager.run(subexperiments)
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:420: in wrapper
    return super().run(
.tox/py312/lib/python3.12/site-packages/qiskit/passmanager/passmanager.py:246: in run
    return meth(*meth_args, **meth_kwargs)
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:182: in run
    return parallel_map(
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:171: in parallel_map
    return super().run(
.tox/py312/lib/python3.12/site-packages/qiskit/passmanager/passmanager.py:246: in run
    raise error
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:161: in parallel_map
    return parallel_map(
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:171: in parallel_map
    future = executor.map(_task_wrapper, param)
/usr/local/lib/python3.12/concurrent/futures/process.py:822: in map
    raise error
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:161: in parallel_map
    results = super().map(partial(_process_chunk, fn),
/usr/local/lib/python3.12/concurrent/futures/_base.py:608: in map
    future = executor.map(_task_wrapper, param)
/usr/local/lib/python3.12/concurrent/futures/process.py:822: in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
/usr/local/lib/python3.12/concurrent/futures/process.py:794: in submit
    results = super().map(partial(_process_chunk, fn),
/usr/local/lib/python3.12/concurrent/futures/_base.py:608: in map
    self._start_executor_manager_thread()
/usr/local/lib/python3.12/concurrent/futures/process.py:733: in _start_executor_manager_thread
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
/usr/local/lib/python3.12/concurrent/futures/process.py:794: in submit
    self._launch_processes()
/usr/local/lib/python3.12/concurrent/futures/process.py:760: in _launch_processes
    self._start_executor_manager_thread()
/usr/local/lib/python3.12/concurrent/futures/process.py:733: in _start_executor_manager_thread
    self._spawn_process()
/usr/local/lib/python3.12/concurrent/futures/process.py:770: in _spawn_process
    self._launch_processes()
/usr/local/lib/python3.12/concurrent/futures/process.py:760: in _launch_processes
    p.start()
/usr/local/lib/python3.12/multiprocessing/process.py:121: in start
    self._spawn_process()
/usr/local/lib/python3.12/concurrent/futures/process.py:770: in _spawn_process
    self._popen = self._Popen(self)
/usr/local/lib/python3.12/multiprocessing/context.py:282: in _Popen
    p.start()
/usr/local/lib/python3.12/multiprocessing/process.py:121: in start
    return Popen(process_obj)
/usr/local/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
    self._popen = self._Popen(self)
/usr/local/lib/python3.12/multiprocessing/context.py:282: in _Popen
    self._launch(process_obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <multiprocessing.popen_fork.Popen object at 0x7fafdf377710>
process_obj = <ForkProcess name='ForkProcess-1' parent=6471 unknown>

    return Popen(process_obj)
/usr/local/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
    def _launch(self, process_obj):
        code = 1
        parent_r, child_w = os.pipe()
        child_r, parent_w = os.pipe()
>       self.pid = os.fork()

which points to exactly the line that is hanging under nbmake in the tests under Python 3.8.

(I wonder if the issue here might be in part due to interaction with pytest and/or nbmake, but it's doesn't appear like nbmake uses any multiprocessing directly.)

So perhaps this is again an instance of https://pythonspeed.com/articles/python-multiprocessing/, in which it could perhaps be fixed by using mp.get_context("spawn") in Qiskit.

Possibly relevant issues

garrison commented 6 months ago

Another way we could potentially work around this is by setting QISKIT_NUM_PROCS=1.

garrison commented 6 months ago

The symptom seems to have been fixed by performing the ubuntu tests on Python 3.9 instead of 3.8. With #556, the generate_cutting_experiments function will no longer use the pass manager (and therefore not need multiprocessing), so we should no longer see this even on Python 3.8 once that PR is merged.