search

dask-contrib / dask-awkward

Native Dask collection for awkward arrays, and the library to use it.
https://dask-awkward.readthedocs.io
BSD 3-Clause "New" or "Revised" License
61 stars 19 forks source link

Misleading cupy error instead of AxisError #532

Closed alexander-held closed 3 months ago

alexander-held commented 3 months ago

During debugging I ran into the following:

import awkward as ak
import dask_awkward as dak

arr = ak.Array([[1,3,4], 5])
ak.flatten(arr, axis=1)

rightfully fails with

AxisError: axis=1 exceeds the depth of this array (1)

This error occurred while calling

    ak.flatten(
        <Array [[1, 3, 4], 5] type='2 * union[var * int64, int64]'>
        axis = 1
    )

and could be fixed with arr = ak.Array([[1,3,4], [5]]). The dask-awkward version however

arr = dak.from_awkward(ak.Array([[1,3,4], 5]), npartitions=1)
dak.flatten(arr)

fails with

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File /usr/local/lib/python3.10/site-packages/awkward/_singleton.py:27, in PrivateSingleton._ensure_instance(cls)
     26 try:
---> 27     return cls._instance
     28 except AttributeError:

AttributeError: type object 'Cupy' has no attribute '_instance'

During handling of the above exception, another exception occurred:

[...]

ModuleNotFoundError: to use Awkward Arrays with CUDA, you must install cupy:

    pip install cupy

or

    conda install -c conda-forge cupy

This error occurred while calling

    ak.flatten(
        <Array-typetracer [...] type='## * union[var * int64, int64]'>
        axis = 1
        highlevel = True
        behavior = None
        attrs = None
    )

I'm not sure where the CUDA connection comes from but I assume something is going wrong here to end up with it.

I'm using

dak.__version__, ak.__version__
('2024.7.0', '2.6.7')
martindurant commented 3 months ago

The error happens while calling ak.flatten on the typetracer metadata array.

Awkward-only version:

arr = ak.Array([[1,3,4], 5])
tarr = arr.layout.to_typetracer()
ak.flatten(tarr)

I suggest we transfer this issue to awkward.

alexander-held commented 3 months ago

Thanks for narrowing it down, I don't have permissions to transfer so I imagine we need @jpivarski for that perhaps?

jpivarski commented 3 months ago

I saw that before: someone raised this as an issue, I dug into it, and it was an already-fixed issue that you can get just by upgrading. I just tried to reproduce

>>> arr = ak.Array([[1,3,4], 5])
>>> tarr = arr.layout.to_typetracer()
>>> ak.flatten(tarr)

and the error message is appropriate:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/irishep/awkward/src/awkward/_dispatch.py", line 64, in dispatch
    next(gen_or_result)
  File "/home/jpivarski/irishep/awkward/src/awkward/operations/ak_flatten.py", line 172, in flatten
    return _impl(array, axis, highlevel, behavior, attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jpivarski/irishep/awkward/src/awkward/operations/ak_flatten.py", line 236, in _impl
    out = ak._do.flatten(layout, axis)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jpivarski/irishep/awkward/src/awkward/_do.py", line 196, in flatten
    offsets, flattened = layout._offsets_and_flattened(axis, 1)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jpivarski/irishep/awkward/src/awkward/contents/unionarray.py", line 884, in _offsets_and_flattened
    offsets, flattened = self._contents[i]._offsets_and_flattened(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jpivarski/irishep/awkward/src/awkward/contents/numpyarray.py", line 446, in _offsets_and_flattened
    raise AxisError(f"axis={axis} exceeds the depth of this array ({depth})")
numpy.exceptions.AxisError: axis=1 exceeds the depth of this array (1)

This error occurred while calling

    ak.flatten(
        <Array-typetracer [...] type='2 * union[var * int64, int64]'>
    )

I'm having trouble finding where I had this conversation before (maybe a week or two ago), so I'm not able to link to it, but it's a solved problem.

In fact, I'm going to close this issue because it's straightforward.

alexander-held commented 3 months ago

Which version did you upgrade to? With awkward 2.6.7, which is the latest release (and I see no commits to main since then), I can reproduce this error still.

jpivarski commented 3 months ago

I just ensured that I have Awkward 2.6.7 installed from PyPI (not a development copy) and I get the AxisError above.

alexander-held commented 3 months ago

I'm not sure what the difference in our setups is. Here is a clean reproducer in a container:

~ ❯ docker run -it --rm python:3.11-slim /bin/bash
root@dc195deba2ab:/# pip install awkward
Collecting awkward
  Obtaining dependency information for awkward from https://files.pythonhosted.org/packages/54/0a/aba724626176df414fe9c4bc55f532cf951a66141243f2bae40e9315896c/awkward-2.6.7-py3-none-any.whl.metadata
  Downloading awkward-2.6.7-py3-none-any.whl.metadata (7.0 kB)
Collecting awkward-cpp==37 (from awkward)
  Obtaining dependency information for awkward-cpp==37 from https://files.pythonhosted.org/packages/05/e9/4fe8c2c8de344c24639f8363c36af04c6f3f510be49572e5c5659729adf9/awkward_cpp-37-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading awkward_cpp-37-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (2.2 kB)
Collecting fsspec>=2022.11.0 (from awkward)
  Obtaining dependency information for fsspec>=2022.11.0 from https://files.pythonhosted.org/packages/5e/44/73bea497ac69bafde2ee4269292fa3b41f1198f4bb7bbaaabde30ad29d4a/fsspec-2024.6.1-py3-none-any.whl.metadata
  Downloading fsspec-2024.6.1-py3-none-any.whl.metadata (11 kB)
Collecting importlib-metadata>=4.13.0 (from awkward)
  Obtaining dependency information for importlib-metadata>=4.13.0 from https://files.pythonhosted.org/packages/82/47/bb25ec04985d0693da478797c3d8c1092b140f3a53ccb984fbbd38affa5b/importlib_metadata-8.2.0-py3-none-any.whl.metadata
  Downloading importlib_metadata-8.2.0-py3-none-any.whl.metadata (4.7 kB)
Collecting numpy>=1.18.0 (from awkward)
  Obtaining dependency information for numpy>=1.18.0 from https://files.pythonhosted.org/packages/72/44/71ac0090d4ccb512fcac0ef0e5208248423a1ce30381541700470ac09b75/numpy-2.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading numpy-2.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.0/62.0 kB 3.0 MB/s eta 0:00:00
Collecting packaging (from awkward)
  Obtaining dependency information for packaging from https://files.pythonhosted.org/packages/08/aa/cc0199a5f0ad350994d660967a8efb233fe0416e4639146c089643407ce6/packaging-24.1-py3-none-any.whl.metadata
  Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting zipp>=0.5 (from importlib-metadata>=4.13.0->awkward)
  Obtaining dependency information for zipp>=0.5 from https://files.pythonhosted.org/packages/20/38/f5c473fe9b90c8debdd29ea68d5add0289f1936d6f923b6b9cc0b931194c/zipp-3.19.2-py3-none-any.whl.metadata
  Downloading zipp-3.19.2-py3-none-any.whl.metadata (3.6 kB)
Downloading awkward-2.6.7-py3-none-any.whl (827 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 827.4/827.4 kB 16.0 MB/s eta 0:00:00
Downloading awkward_cpp-37-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (582 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 582.4/582.4 kB 47.9 MB/s eta 0:00:00
Downloading fsspec-2024.6.1-py3-none-any.whl (177 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.6/177.6 kB 55.6 MB/s eta 0:00:00
Downloading importlib_metadata-8.2.0-py3-none-any.whl (25 kB)
Downloading numpy-2.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (13.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/13.9 MB 55.0 MB/s eta 0:00:00
Downloading packaging-24.1-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 kB 18.1 MB/s eta 0:00:00
Downloading zipp-3.19.2-py3-none-any.whl (9.0 kB)
Installing collected packages: zipp, packaging, numpy, fsspec, importlib-metadata, awkward-cpp, awkward
Successfully installed awkward-2.6.7 awkward-cpp-37 fsspec-2024.6.1 importlib-metadata-8.2.0 numpy-2.0.1 packaging-24.1 zipp-3.19.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.2.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
root@dc195deba2ab:/# python
Python 3.11.7 (main, Dec 19 2023, 10:35:38) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import awkward as ak
>>> arr = ak.Array([[1,3,4], 5])
>>> tarr = arr.layout.to_typetracer()
>>> ak.flatten(tarr)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/awkward/_singleton.py", line 27, in _ensure_instance
    return cls._instance
           ^^^^^^^^^^^^^
AttributeError: type object 'Cupy' has no attribute '_instance'. Did you mean: 'instance'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/awkward/_dispatch.py", line 64, in dispatch
    next(gen_or_result)
  File "/usr/local/lib/python3.11/site-packages/awkward/operations/ak_flatten.py", line 172, in flatten
    return _impl(array, axis, highlevel, behavior, attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/operations/ak_flatten.py", line 236, in _impl
    out = ak._do.flatten(layout, axis)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/_do.py", line 196, in flatten
    offsets, flattened = layout._offsets_and_flattened(axis, 1)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/contents/unionarray.py", line 887, in _offsets_and_flattened
    offsetsraws[i] = offsets.ptr
                     ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/index.py", line 141, in ptr
    elif self._nplike == Cupy.instance():
                         ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/_singleton.py", line 47, in instance
    return cls._ensure_instance()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/_singleton.py", line 29, in _ensure_instance
    return cls._new()
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/_singleton.py", line 19, in _new
    self.__init__()  # pylint: disable=unnecessary-dunder-call
    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/_nplikes/cupy.py", line 27, in __init__
    self._module = ak._connect.cuda.import_cupy("Awkward Arrays with CUDA")
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/awkward/_connect/cuda/__init__.py", line 194, in import_cupy
    raise ModuleNotFoundError(error_message.format(name))
ModuleNotFoundError: to use Awkward Arrays with CUDA, you must install cupy:

    pip install cupy

or

    conda install -c conda-forge cupy

This error occurred while calling

    ak.flatten(
        <Array-typetracer [...] type='2 * union[var * int64, int64]'>
    )
>>>
root@dc195deba2ab:/# exit
exit
jpivarski commented 3 months ago

In your /usr/local/lib/python3.11/site-packages/awkward/contents/unionarray.py, it got past line 884 (calling _offsets_and_flattened) and got to line 887 (where it tries to load offsets.ptr) in

https://github.com/scikit-hep/awkward/blob/3ab54f8daec75ef4847562595fdeb064b25f2a69/src/awkward/contents/unionarray.py#L882-L890

In my copy, it failed on line 884 because this goes to line 451 of

https://github.com/scikit-hep/awkward/blob/3ab54f8daec75ef4847562595fdeb064b25f2a69/src/awkward/contents/numpyarray.py#L442-L451

Hmm. But it must be the second time around the for loop in the unionarray.py code snippet where it failed; you probably see a difference between

ak.Array([[1,3,4], 5])

and

ak.Array([5, [1,3,4]])

because it would have caught the AxisError on the UnionArray's first content.

Your problem is that offsets.ptr even attempts to try to use CuPy. The discrepancy must be in

https://github.com/scikit-hep/awkward/blob/3ab54f8daec75ef4847562595fdeb064b25f2a69/src/awkward/index.py#L137-L142

which, for me, checks to see if the array is NumPy or CuPy before deciding how to get its pointer. Before #3086 (4 months ago), it just assumed CuPy, as in

https://github.com/scikit-hep/awkward/blob/21af5dccad55a8b9e447b7be4af8b35226f1f95c/src/awkward/index.py#L137-L139

But that fix has been in every version since 2.6.4.

If you look in your /usr/local/lib/python3.11/site-packages/awkward/ directory, does any of the code you're running not look like the GitHub snippets above? The discrepancy between what you're running and what I'm running is very localized; it has to be in or near one of these snippets.

alexander-held commented 3 months ago

I can confirm that ak.Array([5, [1,3,4]]) results in

numpy.exceptions.AxisError: axis=1 exceeds the depth of this array (1)

This error occurred while calling

    ak.flatten(
        <Array-typetracer [...] type='2 * union[int64, var * int64]'>
    )

instead of some cupy message.

https://github.com/scikit-hep/awkward/blob/3ab54f8daec75ef4847562595fdeb064b25f2a69/src/awkward/index.py#L137-L142 looks the exact same for me in my container as the code on GitHub. Can you reproduce the cupy error within a container? Is there any possibly ARM-specific code path here that might matter? I am running this on a M1 at the moment, but saw the original error on what I assume was some x86.

jpivarski commented 3 months ago

There shouldn't be anything Mac-specific about it, but I just reproduced it on a Mac.

Found it: the difference is being in an environment without CuPy installed. The code that checks to see what kind of nplike you have,

https://github.com/scikit-hep/awkward/blob/3ab54f8daec75ef4847562595fdeb064b25f2a69/src/awkward/index.py#L137-L142

actually creates the Cupy singleton, which, in turn, imports CuPy.

This fixes it: scikit-hep/awkward#3206.

alexander-held commented 3 months ago

Thanks for tracking this down!

jpivarski commented 3 months ago

(The lack of an else clause on the original function made me nervous, anyway. If it hadn't been an error case, the TypeTracer wouldn't have worked anyway. The second test in the PR addresses that.)