search

dask / community

For general discussion and community planning. Discussion issues welcome.
20 stars 3 forks source link

Manage dependencies with poetry? #203

Closed gjoseph92 closed 2 years ago

gjoseph92 commented 2 years ago

Surely this has come up before, but I couldn't find an issue for it.

Just curious if there's been discussion about using a deterministic, locking package manager like poetry for dask/dask and dask/distributed.

Just thought of it today with the typical spurt of CI failures due to upstream packages releasing new versions: https://github.com/dask/dask/issues/8366 https://github.com/dask/dask/issues/8364 https://github.com/dask/dask/issues/8362

For contributors, it's nice to not have your PR suddenly go red, and be blocked until someone (like @jsignell) makes and merges the changes to work with (or temporarily pin) the new version of a dependency. As a contributor, I also prefer the development experience with a package manager like poetry, since you don't even have to read any "set up your development environment" docs—just run the same command for any poetry project (poetry install --dev) and get the exact same environment as CI.

Of course, we'd still test against new versions of upstream packages. It's just nice when the changes happen deterministically, on your terms (only when you poetry update).

There are other downsides: mindeps isn't automatically supported (https://github.com/python-poetry/poetry/issues/3527), and incompatibilities between dependencies are hard to work around (https://github.com/python-poetry/poetry/issues/697). But just wondering if there's any interest in this.

jsignell commented 2 years ago

To be fair dask/dask#8366 is really just me neglecting upstream failures for several weeks and that action doesn't run on every pull request so it doesn't block anyone.

I am a little worried that if we didn't have so much red CI when an upstream package releases something breaking, it would be easier to miss. But I suppose if we had a nightly job that we closely monitored, then we'd be able to stay on top of released packages that break us.

jakirkham commented 2 years ago

Honestly would rather avoid it. We've had quite a lot of pain in the Conda packaging community with Poetry.

jrbourbeau commented 2 years ago

Yeah, I agree with @jsignell and @jakirkham -- I'd rather focus on increasing visibility/attention of our existing upstream CI build

jsignell commented 2 years ago

I'm going to close this for now butt thank you for bringing it up Gabe. It is always good to step back a bit and make sure we are happy with what we are doing :)