search

dask-contrib / dask-awkward

Native Dask collection for awkward arrays, and the library to use it.
https://dask-awkward.readthedocs.io
BSD 3-Clause "New" or "Revised" License
61 stars 19 forks source link

Circular import when using latest `dask` #544

Closed GaetanLepage closed 2 months ago

GaetanLepage commented 2 months ago

Hello, dask(-awkward) nixpkgs maintainer here.

I am facing an issue when trying to bump the versions of the dask-* packages to 2024.9.0. Although dask-awkward hasn't had a new release since 2024.7.0, it now fails to import with:

Check whether the following modules can be imported: dask_awkward
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <lambda>
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/nix/store/cd7hs8madz5gpm92b08sdg3gjvmmj7df-python3.12-dask-awkward-2024.7.0/lib/python3.12/site-packages/dask_awkward/__init__.py", line 1, in <module>
    from dask_awkward import config  # isort:skip; load awkward config
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/cd7hs8madz5gpm92b08sdg3gjvmmj7df-python3.12-dask-awkward-2024.7.0/lib/python3.12/site-packages/dask_awkward/config.py", line 3, in <module>
    import dask.config
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/__init__.py", line 5, in <module>
    from dask.base import (
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/base.py", line 1039, in <module>
    from dask import multiprocessing as dask_multiprocessing
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/multiprocessing.py", line 19, in <module>
    from dask.optimization import cull, fuse
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/optimization.py", line 10, in <module>
    from dask._task_spec import GraphNode
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/_task_spec.py", line 90, in <module>
    from dask.sizeof import sizeof
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/sizeof.py", line 318, in <module>
    _register_entry_point_plugins()
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/sizeof.py", line 309, in _register_entry_point_plugins
    registrar = entry_point.load()
                ^^^^^^^^^^^^^^^^^^
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ynfzh0ncgpja9sy9wj6x25c09c4vi314-python3.12-dask-histogram-2024.3.0/lib/python3.12/site-packages/dask_histogram/__init__.py", line 6, in <module>
    from dask_histogram.core import (
  File "/nix/store/ynfzh0ncgpja9sy9wj6x25c09c4vi314-python3.12-dask-histogram-2024.3.0/lib/python3.12/site-packages/dask_histogram/core.py", line 13, in <module>
    from dask.blockwise import BlockwiseDep, blockwise, fuse_roots, optimize_blockwise
  File "/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/blockwise.py", line 13, in <module>
    from dask.base import clone_key, get_name_from_key, tokenize
ImportError: cannot import name 'clone_key' from partially initialized module 'dask.base' (most likely due to a circular import) (/nix/store/gw417fz2s7y05vyfy55m3kbgz42s9hfp-python3.12-dask-2024.9.0/lib/python3.12/site-packages/dask/base.py)

Any idea ?

lgray commented 2 months ago

This appears to be something that changed in the most recent release of dask that was not communicated to any contrib packages. You can see more here: https://github.com/dask-contrib/dask-histogram/issues/149

@martindurant any idea?

GaetanLepage commented 2 months ago

Thanks @martindurant for solving this rapidly !