search

radio-astro-tools / spectral-cube

Library for reading and analyzing astrophysical spectral data cubes
http://spectral-cube.rtfd.org
BSD 3-Clause "New" or "Revised" License
95 stars 62 forks source link

no attribute __spec__ ? #791

Open keflavich opened 2 years ago

keflavich commented 2 years ago

I've had sporadic errors like this:

Traceback (most recent call last):
  File "/orange/adamginsburg/ALMA_IMF/reduction/analysis/statcont_cubes.py", line 123, in <module>
    noise = cube.std()
  File "/blue/adamginsburg/adamginsburg/repos/spectral-cube/spectral_cube/dask_spectral_cube.py", line 116, in wrapper
    out = function(self, *args, **kwargs)
  File "/blue/adamginsburg/adamginsburg/repos/spectral-cube/spectral_cube/dask_spectral_cube.py", line 67, in wrapper
    return function(self, *args, **kwargs)
  File "/blue/adamginsburg/adamginsburg/repos/spectral-cube/spectral_cube/dask_spectral_cube.py", line 687, in std
    return self._compute(da.nanstd(self._get_filled_data(fill=np.nan), axis=axis, ddof=ddof, **kwargs))
  File "/blue/adamginsburg/adamginsburg/repos/spectral-cube/spectral_cube/dask_spectral_cube.py", line 288, in _compute
    return array.compute(**self._scheduler_kwargs)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/dask/base.py", line 288, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/dask/base.py", line 570, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/dask/multiprocessing.py", line 219, in get
    result = get_async(
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/dask/local.py", line 495, in get_async
    fire_tasks(chunksize)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/dask/local.py", line 490, in fire_tasks
    fut = submit(batch_execute_tasks, each_args)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/concurrent/futures/process.py", line 697, in submit
    self._adjust_process_count()
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/concurrent/futures/process.py", line 675, in _adjust_process_count
    p.start()
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/multiprocessing/spawn.py", line 183, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'

this looks like a problem somewhere in dask? it could just be what happens when a dask thread gets terminated badly, I'm not sure. It's probably upstream but I'm posting it here because we may get other confused users coming this way.

keflavich commented 2 years ago

This might have been caused by using the multiprocessing scheduler