search

dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.57k stars 717 forks source link

Upload of `distributed-impl` fails due to duplicate #5864

Closed jakirkham closed 2 years ago

jakirkham commented 2 years ago

Seeing the following on CI:

Uploading file "dask/distributed-impl/2022.02.1a220224/noarch/distributed-impl-2022.02.1a220224-cython.tar.bz2"
Distribution already exists. Please use the -i/--interactive or --force or --skip options or `anaconda remove dask/distributed-impl/2022.02.1a220224/noarch/distributed-impl-2022.02.1a220224-cython.tar.bz2
Error:  ('file noarch/distributed-impl-2022.02.1a220224-cython.tar.bz2 already exists for package distributed-impl version 2022.02.1a220224', 409)
Error: Process completed with exit code 1.

cc @charlesbluca

jakirkham commented 2 years ago

Wondering if we should start including a time stamp or git hash to help distinguish these further

charlesbluca commented 2 years ago

Good catch, completely forgot that these are only distinguished by the pre-release version

Wondering if we should start including a time stamp or git hash to help distinguish these further

I think the hash makes sense here to distinguish which distributed commit triggered the build, though it'd probably be important to note that would change the install command for specifying Cython slightly from

conda install -c dask/label/dev distributed-impl=*=cython

to

conda install -c dask/label/dev distributed-impl=*=*cython
charlesbluca commented 2 years ago

Alternatively we could add the --force flag to the noarch uploads, as I don't think there's a chance for simultaneous uploads here

jakirkham commented 2 years ago

Challenge with --force is we lose reproducibility (say if the previous package showed some issue we wanted to reproduce)

jakirkham commented 2 years ago

We might be able include the hash in the version. For example using the + syntax that versioneer often uses for this kind of thing

jakirkham commented 2 years ago

In conda-forge we have a package that uses a full timestamp. So maybe an option if we go that way

charlesbluca commented 2 years ago

In that case, let's go with the hash for now as a minimal solution - if for whatever reason that doesn't work out, happy to switch to a full timestamp

charlesbluca commented 2 years ago

Opened https://github.com/dask/distributed/pull/5865 to resolve this / continue the conversation

jakirkham commented 2 years ago

Fixed in PR ( https://github.com/dask/distributed/pull/5865 )

jakirkham commented 2 years ago

Seems to have worked 😀