Error multiprocessing + spawn in Python 3.8 on macOS

[silexanaej]

Hi, It seems that there is an error when xrayutilities (I am using v1.7.1) runs with Python 3.8 on macOS (10.14.6 in my case).

Here is a minimal example when simulating a powder diffraction pattern:

import xrayutilities as xu
import numpy as np

wl = 1.85
tth_range = np.linspace(0, 180, 1800)
powder = xu.simpack.smaterials.Powder(xu.materials.Fe, 1)
pm = xu.simpack.PowderModel(powder, wl=wl)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/jahernan/anaconda3/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/jahernan/anaconda3/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/jahernan/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/jahernan/Documents/2021/LULI2000_XRD_Fe/Box_model/Test_XU.py", line 7, in <module>
    pm = xu.simpack.PowderModel(powder, wl=wl)
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/simpack/powdermodel.py", line 81, in __init__
    self.pdiff.append(PowderDiffraction(mat, **kwargs))
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/simpack/powder.py", line 1871, in __init__
    self._init_multiprocessing()
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/simpack/powder.py", line 1902, in _init_multiprocessing
    mg.start()
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/managers.py", line 579, in start
    self._process.start()
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "Test_XU.py", line 7, in <module>
    pm = xu.simpack.PowderModel(powder, wl=wl)
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/simpack/powdermodel.py", line 81, in __init__
    self.pdiff.append(PowderDiffraction(mat, **kwargs))
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/simpack/powder.py", line 1871, in __init__
    self._init_multiprocessing()
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/simpack/powder.py", line 1902, in _init_multiprocessing
    mg.start()
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/Users/jahernan/anaconda3/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

It is related to the use of multiprocessing in 'spawn' mode. This is the default on Python 3.8 on macOS whereas the 'fork' method was the default in Python 3.7.

Setting the 'fork' method (multiprocessing.set_start_method('fork')) in powder.py in the function _init_multiprocessing() solves the problem but it should be set everywhere multiprocessing is used:

def _init_multiprocessing(self):
        """
        initialize multiprocessing for powder pattern calculation
        """
        # The structure of the multiprocessing code is as follows:
        # There are nproc "manager"s which handle the actual convolver code.
        # Additionally there are 4 daemon threads which listen for work to be
        # distributed to the managers.
        multiprocessing.set_start_method('fork') # HERE, imposing fork method (spawn is default on macOS with python 3.8)
        np = config.NTHREADS
        self.nproc = np if np != 0 else multiprocessing.cpu_count()
        self.chunks = chunkify(list(self.data), self.nproc)
        self.next_proc = len(self.data) % self.nproc
        manager.register("conv", convolver_handler)
        self.managers = [manager() for idx in range(self.nproc)]
        self.conv_handlers = []
        self.threads = []
        self.output_queue = queue.Queue()
        for idx, mg in enumerate(self.managers):
            mg.start()
            m = mg.conv()
            for h in self.chunks[idx]:
                m.add_convolver(self.data[h]['conv'])
            self.conv_handlers.append(m)
            self.threads.append((
                threading.Thread(target=self._send_work, args=(idx, )),
                queue.Queue(), self.output_queue))
        self._running = True
        for th, q1, q2 in self.threads:
            th.daemon = True
            th.start()
        atexit.register(self.__stop__)

Best regards

[dkriegner]

thanks for reporting this. I have no access to a macOS system.

I am not sure why this is not detected by the unit tests which run this code on macOS with Python 3.9 without error. could you try to confirm that the unit test fails on your configuration? (It could depend on the number of CPUs?)

In the xrayutilties sources folder you can run:

python -m unittest tests/test_simpack_powdermodel.py

The fix you propose seems awkward because the Python documentation mentions that fork should be considered unsafe on macOS. Also fork does not exist on Windows so in xrayutilities one would need to change this setting only for the macOS platform or at least check

if 'fork' in multiprocessing.get_all_start_methods():
    multiprocessing.set_start_method('fork')

[silexanaej]

Running the unit test in the sources folder gives the following error (I also get it just by running "python" in xrayutilities folder):

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
Traceback (most recent call last):
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/io/__init__.py", line 19, in <module>
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/io/cbf.py", line 23, in <module>
ModuleNotFoundError: No module named 'numpy'

Also fork does not exist on Windows so in xrayutilities one would need to change this setting only for the macOS platform or at least check

if 'fork' in multiprocessing.get_all_start_methods(): multiprocessing.set_start_method('fork')

I also noticed that if _init_multiprocessing() is called several times (for instance if we want to compute several powder patterns) multiprocessing complains that the context has already been set: RuntimeError: context has already been set

Forcing the fork method allows to avoid this error:

if 'fork' in multiprocessing.get_all_start_methods():
    multiprocessing.set_start_method('fork', force=True)

but to be honest I have no idea if this is a safe solution...

[dkriegner]

Running the unit test in the sources folder gives the following error (I also get it just by running "python" in xrayutilities folder):

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
Traceback (most recent call last):
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/io/__init__.py", line 19, in <module>
  File "/Users/jahernan/anaconda3/lib/python3.8/site-packages/xrayutilities/io/cbf.py", line 23, in <module>
ModuleNotFoundError: No module named 'numpy'

Here you seem to use a different python interpreter (maybe the system python) which does not have access to all the python packages needed for xrayutilities. Can you try to run this with your anaconda python version?

I must omit I am hesitant to mess around with the multiprocessing settings because this is somehow a fragile part of the code and I am afraid to break more then we fix here. I would like to understand first why this is not detected by the unittests on macOS

[silexanaej]

Here you seem to use a different python interpreter (maybe the system python) which does not have access to all the python packages needed for xrayutilities. Can you try to run this with your anaconda python version?

I am actually running the anaconda version, and I have no problem in running it within other folders. It fails only when I run it within the xrayutilities folder.

I must omit I am hesitant to mess around with the multiprocessing settings because this is somehow a fragile part of the code and I am afraid to break more then we fix here. I would like to understand first why this is not detected by the unittests on macOS

Yes of course.

[dkriegner]

Here you seem to use a different python interpreter (maybe the system python) which does not have access to all the python packages needed for xrayutilities. Can you try to run this with your anaconda python version?

I am actually running the anaconda version, and I have no problem in running it within other folders. It fails only when I run it within the xrayutilities folder.

The shown error has clearly nothing to do with xrayutilities, but seems related to the used python. This python has no access to the numpy package. Since you were running xrayutilities (which requires numpy) before I still believe this is an issue with the interpreter.

Anyways I found in the meantime that for the mentioned test (tests/test_simpack_powdermodel.py) to run you would need the additional testfiles available at https://sourceforge.net/projects/xrayutilities/files/ . If you are willing to debug this further please go ahead and check also the tests/README.txt file.

I will see if I can get physical access to a macOS system to check this myself

[silexanaej]

The shown error has clearly nothing to do with xrayutilities, but seems related to the used python. This python has no access to the numpy package. Since you were running xrayutilities (which requires numpy) before I still believe this is an issue with the interpreter.

It seems that it is because I cannot run python properly from a directory that contains folder names that are similar to python modules such as io or math.

Anyway, I was able to run the tests without any errors:

python -m unittest discover

Here are the last outputted lines:

----------------------------------------------------------------------
Ran 244 tests in 758.849s

OK

Idem if I just run

python -m unittest test_simpack_powdermodel.py

----------------------------------------------------------------------
Ran 3 tests in 18.043s

OK

[dkriegner]

Just to be sure: The tests where run using the xrayutilities code which produced the originally reported error?

I do not really understand why your example script behaves different then the test script.

[silexanaej]

Yes I have just checked, same python and xrayutilities versions. However, I noticed something:

In test_simpack_powdermodel.py xu.simpack.PowderModel() is called within a function named setUp().

If I run python -m unittest test_simpack_powdermodel.py or python test_simpack_powdermodel.pyit runs the tests without problems.

I can make test_simpack_powdermodel.py fail and reproduce my error if I call xu.simpack.PowderModel() out of the function (see below just before setUp()) AND if I run it as python test_simpack_powdermodel.py.

import os
import unittest
from multiprocessing import freeze_support

import numpy
import xrayutilities as xu

try:
    import lmfit
except ImportError:
    lmfit = None

testfile = 'LaB6_d500_si_psd.xye.bz2'
datadir = os.path.join(os.path.dirname(__file__), 'data')
fullfilename = os.path.join(datadir, testfile)

@unittest.skipIf(not os.path.isfile(fullfilename) or lmfit is None,
                 "additional test data (see http://xrayutilities.sf.io) and "
                 "the lmfit Python package are needed")
class Test_PowderModel(unittest.TestCase):
    chi2max = 1.5
    # define powder material
    La = xu.materials.elements.La
    B = xu.materials.elements.B
    LaB6 = xu.materials.Crystal(
        "LaB6", xu.materials.SGLattice(221, 4.15692, atoms=[La, B],
                                       pos=['1a', ('6f', 0.19750)],
                                       b=[0.05, 0.15]))
    LaB6_powder = xu.simpack.Powder(LaB6, 1,
                                    crystallite_size_gauss=1e6,
                                    crystallite_size_lor=0.5e-6,
                                    strain_gauss=0,
                                    strain_lor=0)

    # machine settings
    settings = {'classoptions': {'oversampling': 10},
                'global': {'diffractometer_radius': 0.337,
                           'equatorial_divergence_deg': 0.40},
                'tube_tails': {'tail_left': -0.001,
                               'main_width': 0.00015,
                               'tail_right': 0.001,
                               'tail_intens': 0.0015},
                'axial': {'angI_deg': 2.0, 'angD_deg': 2.0,
                          'slit_length_target': 0.008,
                          'n_integral_points': 21,
                          'length_sample': 0.015,
                          'slit_length_source': 0.008001},
                'si_psd': {'si_psd_window_bounds': (0, 32e-3)},
                'absorption': {'sample_thickness': 500e-6,
                               'absorption_coefficient': 3e4},
                'displacement': {'specimen_displacement': -3.8e-5,
                                 'zero_error_deg': 0.0},
                'emission': {'emiss_intensities': (1.0, 0.45)}}
    # define background
    btt, bint = numpy.asarray([(15.158, 1136.452),
                               (17.886, 841.925),
                               (22.906, 645.784),
                               (26.556, 551.663),
                               (34.554, 401.219),
                               (45.764, 260.595),
                               (58.365, 171.993),
                               (81.950, 112.838),
                               (92.370, 101.276),
                               (106.441, 102.486),
                               (126.624, 112.838),
                               (139.096, 132.063),
                               (146.240, 136.500),
                               (152.022, 157.204)]).T

    pm = xu.simpack.PowderModel(LaB6_powder, I0=1.10e6, fpsettings=settings) # HERE, not in the original test but allows to reproduce the error

    def setUp(self):
        with xu.io.xu_open(fullfilename) as fid:
            self.tt, self.det, self.sig = numpy.loadtxt(fid, unpack=True)
        self.mask = numpy.logical_and(self.tt > 18, self.tt < 148)
        self.pm = xu.simpack.PowderModel(self.LaB6_powder, I0=1.10e6,
                                         fpsettings=self.settings)
        self.pm.set_background('spline', x=self.btt, y=self.bint)

But if I run it as python -m unittest test_simpack_powdermodel.py it runs without error.

Of course, if I force the 'fork' method it runs without error in all cases.

Is this of any help?

[dkriegner]

After your last report it starts to get clearer: It seems to be a manifestation of this behavior: https://stackoverflow.com/questions/60691363/runtimeerrorfreeze-support-on-mac

As you see in the example script (and equally in the discussed test script) the PowderModel code is only run inside the main function.

On linux (or to be more precise with the fork method) this seems to be no issue. Windows and Mac will need the encapsulation of the multiprocessing code if the default spawn method is used. On Windows in addition the freeze_support() function must be called.

So I think this is not exactly a bug in xrayutilities but default python behavior. I shall include a dedicated note about this in the example (and in the documentation). In fact a note about it is already present for MS Windows systems and just needs to be expanded to mention also macOS

[dkriegner]

I now found this also in the Python documentation: https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods

[silexanaej]

I see. This is a bit incovenient but this is indeed not a bug in xrayutilities. I guess Anyway, thanks for your work!

[dkriegner]

agreed, this is somewhat unfortunate.

I do, however, by no code change in xrayutilities this problem can be avoided. I hope its now clear from the example and documentation. If there are concrete suggestions on more documentation/comments to be added please reopen.