rapidsai / dask-build-environment

Build environments for various dask related projects on gpuCI
4 stars 11 forks source link

Change numpy pinning to 1.20 #18

Closed charlesbluca closed 2 years ago

charlesbluca commented 2 years ago

As of https://github.com/rapidsai/cudf/pull/9687, the latest cuDF / dask-cudf nightlies implicitly require numpy >=1.17,<1.21; this changes our pinning to 1.20 to allow us to grab the latest nightlies again.

cc @pentschev @quasiben

quasiben commented 2 years ago

I'm approving so we can unblcok CI but I think we should probably remove the pin or set a minimum instead to 1.20.1 rather a true pin

charlesbluca commented 2 years ago

That makes sense! If this unblocks CI happy to turn this into a constraint

pentschev commented 2 years ago

Yes, I agree, we should set a minimum of NumPy 1.20. I don't see it, where's the implicit pinning to <1.21 coming from in https://github.com/rapidsai/cudf/pull/9687 ?

charlesbluca commented 2 years ago

We bumped numba's minimum version to 0.54, which places additional constraints on Numpy:

$ mamba create -n test -c rapidsai-nightly -c conda-forge "numba>=0.54" numpy=1.21
...
Encountered problems while solving:
  - package numba-0.54.1-py37h51133e4_0 has constraint numpy >=1.17,<1.21 conflicting with numpy-1.21.0-py37h038b26d_0

Versus the older version:

$ mamba create -n test -c rapidsai-nightly -c conda-forge "numba>=0.53.1" numpy=1.21
...
  + numba                0.53.1  py39h56b8d98_1       conda-forge/linux-64        4 MB
  + numpy                1.21.4  py39hdbf815f_0       conda-forge/linux-64        6 MB
pentschev commented 2 years ago

So Numba 0.53 had no constraint on NumPy maximum version, but 0.54 has? Looks a bit odd, but seems due to Python 3.10 support: https://github.com/numba/numba/pull/7563 .

esc commented 2 years ago

So Numba 0.53 had no constraint on NumPy maximum version, but 0.54 has? Looks a bit odd, but seems due to Python 3.10 support: numba/numba#7563 .

I think that is correct. NumPy pinning was introduced because future versions were causing breakage. IIRC usually the pin for NumPy is created on release branches.

pentschev commented 2 years ago

Yeah, that makes sense, at first it surprised me but now it's all clear. Thanks @esc for the details! 🙂

charlesbluca commented 2 years ago

Looks like the max version here isn't being specified explicitly by Numba either? Inspecting the linux-64 packages, it looks like their max Numpy version is 2.0a0:

image

pentschev commented 2 years ago

I think you only read depends @charlesbluca , as numpy>=1.17,<1.21 is stated in the constraints field.

charlesbluca commented 2 years ago

Ah thanks @pentschev, missed that section 😄 to follow up, this resolved Dask's gpuCI, and with #19 and #20 I switched the Numpy pinning to be a minimum version constraint