rapidsai / cuml

cuML - RAPIDS Machine Learning Library
https://docs.rapids.ai/api/cuml/stable/
Apache License 2.0
4.25k stars 534 forks source link

[NO MRG] Test CI #6047

Closed jakirkham closed 2 months ago

jakirkham commented 2 months ago

For context, one of the test failures we see on CI is

________________ ERROR collecting cuml/tests/test_benchmark.py _________________
test_benchmark.py:21: in <module>
    from cuml.benchmark import datagen, algorithms
/opt/conda/envs/test/lib/python3.11/site-packages/cuml/benchmark/algorithms.py:70: in <module>
    if has_umap():
/opt/conda/envs/test/lib/python3.11/site-packages/cuml/internals/import_utils.py:78: in has_umap
    import umap  # NOQA
/opt/conda/envs/test/lib/python3.11/site-packages/umap/__init__.py:2: in <module>
    from .umap_ import UMAP
/opt/conda/envs/test/lib/python3.11/site-packages/umap/umap_.py:47: in <module>
    from pynndescent import NNDescent
/opt/conda/envs/test/lib/python3.11/site-packages/pynndescent/__init__.py:5: in <module>
    from .pynndescent_ import NNDescent, PyNNDescentTransformer
/opt/conda/envs/test/lib/python3.11/site-packages/pynndescent/pynndescent_.py:23: in <module>
    import pynndescent.sparse_nndescent as sparse_nnd
/opt/conda/envs/test/lib/python3.11/site-packages/pynndescent/sparse_nndescent.py:24: in <module>
    EMPTY_GRAPH = make_heap(1, 1)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/dispatcher.py:442: in _compile_for_args
    raise e
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/dispatcher.py:375: in _compile_for_args
    return_val = self.compile(tuple(argtypes))
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/dispatcher.py:905: in compile
    cres = self._compiler.compile(args, return_type)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/dispatcher.py:80: in compile
    status, retval = self._compile_cached(args, return_type)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/dispatcher.py:94: in _compile_cached
    retval = self._compile_core(args, return_type)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/dispatcher.py:107: in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler.py:744: in compile_extra
    return pipeline.compile_extra(func)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler.py:438: in compile_extra
    return self._compile_bytecode()
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler.py:506: in _compile_bytecode
    return self._compile_core()
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler.py:481: in _compile_core
    raise e
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler.py:472: in _compile_core
    pm.run(self.state)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler_machinery.py:364: in run
    raise e
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler_machinery.py:356: in run
    self._runPass(idx, pass_inst, state)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler_lock.py:35: in _acquire_compile_lock
    return func(*args, **kwargs)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler_machinery.py:311: in _runPass
    mutated |= check(pss.run_pass, internal_state)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/compiler_machinery.py:273: in check
    mangled = func(compiler_state)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typed_passes.py:112: in run_pass
    typemap, return_type, calltypes, errs = type_inference_stage(
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typed_passes.py:93: in type_inference_stage
    errs = infer.propagate(raise_errors=raise_errors)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typeinfer.py:1083: in propagate
    errors = self.constraints.propagate(self)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typeinfer.py:182: in propagate
    raise e
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typeinfer.py:160: in propagate
    constraint(typeinfer)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typeinfer.py:725: in __call__
    attrty = typeinfer.context.resolve_getattr(ty, self.attr)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typing/context.py:283: in resolve_getattr
    attrty = self.resolve_module_constants(typ, attr)
/opt/conda/envs/test/lib/python3.11/site-packages/numba/core/typing/context.py:348: in resolve_module_constants
    attrval = getattr(typ.pymod, attr)
/opt/conda/envs/test/lib/python3.11/site-packages/numpy/__init__.py:397: in __getattr__
    raise AttributeError(
E   AttributeError: `np.infty` was removed in the NumPy 2.0 release. Use `np.inf` instead.

This issue was fixed in upstream PR ( https://github.com/lmcinnes/pynndescent/pull/242 ) and released in 0.5.13

However currently that fix isn't picked up here as pynndescent is pinned to 0.5.5

https://github.com/rapidsai/cuml/blob/973a65ff703a14d9dd8f355af373e2053444b27a/dependencies.yaml#L512

Separately cuML appears not to have a NumPy 2 pin, which causes it to be picked up here. Fixing support with NumPy 2 is being handled in PR ( https://github.com/rapidsai/cuml/pull/6031 ). Also a NumPy pin is being added there to better constrain this dependency

jakirkham commented 2 months ago

Also we are seeing the same hang in pytest cuml-dask CUDA 12.5 wheel tests on CI as we have seen elsewhere: https://github.com/rapidsai/cuml/issues/6050

jakirkham commented 2 months ago

Closing as resolved via PR: https://github.com/rapidsai/cuml/pull/6031