Python Version 3.8

[kvoggel]

Was the zap code already tested with python version 3.8. I am trying to use it and get a persistent error that looks like this :

File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "zap_noskycube.py", line 40, in <module>
    zap.process('DATACUBE_NOSKY.fits', outcubefits='DATACUBE_NOSKY_ZAP_NOMASK.fits', mask='SKYMASK.fits')
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zap/zap.py", line 173, in process
    extSVD = SVDoutput(cubefits, clean=clean, zlevel=zlevel,
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zap/zap.py", line 229, in SVDoutput
    zobj._prepare(clean=clean, zlevel=zlevel, cftype=cftype,
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zap/zap.py", line 283, in wrapped
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zap/zap.py", line 455, in _prepare
    self._zlevel(calctype=zlevel)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zap/zap.py", line 283, in wrapped
    res = func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zap/zap.py", line 579, in _zlevel
    self.zlsky = np.hstack(parallel_map(func, self.stack, NCPU, axis=0))
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zap/zap.py", line 883, in parallel_map
    manager = Manager()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

[saimn]

This could be related to the change of the default start method on macos : https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

But we cannot reproduce this with a Python 3.8.1 from conda on @RolandBacon 's macbook. What is your Python version (3.8.0 ?) and how was it installed ? Could you try with a Python 3.8.1 and/or with conda ?

[kvoggel]

I just double checked and I have Python version 3.8.1 And yes indeed it seems like the change in default multiprocessing might be the issue. But not sure how I can solve that error.

I installed python straight from the source not with conda on my Mac. I just wondered if it was a 3.8 problem as I had just upgraded from python 2.7 to the 3.8 version and many astro packages I installed had issues with the new 3.8 and were only tested in 3.7. Thanks for looking into it though.

[saimn]

Could you check with conda if you can reproduce the issue? I'm sure if we can do something in the code or if this is an issue related to your installation.

[kvoggel]

I made a conda environment with python 3.8.1, scipy 1.4.1 and numpy version 1.18 and it successfully zapped a cube. So indeed it must have been something in my installation of python and the packages. Works with python 3.8 indeed. I've had so many packages recently that did not work under 3.8 I assumed it was the same thing with this error but in fact it wasn't.

[saimn]

Great that it works! I'm not sure if your installation has a problem or if conda does something specific for this issue, but anyway it works so let's close this and see if the issue arise again in the future.