Python kernel crash on parallel RB

[wvlothuizen]

Comment by Jorge:

The problem occurs when one calls the "multiprocessing.Pool.map_async" method for the parallel compilation of the different runs of RB. Unfortunately, there is very little that we know besides this. When we call this method the python kernel crashes silently without raising errors. I have tested sequential compilation of the different runs (without using the multiprocessing tools) and it works fine. Of course this takes an extraordinary amount of time.

[wvlothuizen]

@jorgemfm27: Could you add:

• the actual function you call when the problem occurs
• the commit of PycQED that you use (see git log)

[jorgemfm27]

The function we call is device_object_CCL.measure_two_qubit_randomized_benchmarking. In this method there is a call to the function send_rb_tasks (defined within the method itself) which uses multiprocessing.Pool. This is where the program seems to crash.

We're on commit 1ee32990833721934d932fb940d149144b7864e1

[wvlothuizen]

Testing on my MacBook on the latest version in branch 'develop' proceeds fine, using the recently added test for this particular function (see below). There seem to be no relevant changes since the commit you're using, although upgrading to 'develop' will give you faster RB because of other improvements.

I guess the multiprocessing is the culprit, I'm not sure it has ever been thoroughly tested (I didn't touch it). Maybe try to lower cl_oql.maxtasksperchild from 4 to say 2, and have a look at memory use on your computer using Task Manager (Ctrl-Shift-Esc)

Zaphod-2:PycQED_py3 wouter$ pytest -v pycqed/tests/dev_qubit_objs/test_device_objects.py -k test_measure_two_qubit_randomized_benchmarking
========================================================================================= test session starts =========================================================================================
platform darwin -- Python 3.6.9, pytest-5.4.3, py-1.10.0, pluggy-0.13.1 -- /Users/wouter/.pyenv/versions/3.6.9/bin/python3.6
cachedir: .pytest_cache
rootdir: /Volumes/Data/shared/GIT/PycQED_py3, inifile: pytest.ini
plugins: cov-2.11.1, cookies-0.5.1
collected 24 items / 23 deselected / 1 selected                                                                                                                                                       

pycqed/tests/dev_qubit_objs/test_device_objects.py::Test_Device_obj::test_measure_two_qubit_randomized_benchmarking PASSED                                                                      [100%]

[MiguelSMoreira]

As I understand, randomized benchmarking can now only perform compilation sequentially. Is this issue blocking @jorgemfm27 and/or @hasali92?