search

dask / dask-jobqueue

Deploy Dask on job schedulers like PBS, SLURM, and SGE
https://jobqueue.dask.org
BSD 3-Clause "New" or "Revised" License
235 stars 142 forks source link

Release soon #627

Closed jacobtomlinson closed 9 months ago

jacobtomlinson commented 9 months ago

I was talking to someone earlier who is installing this project from main because they need unreleased changes. There hasn't been a release here since June last year.

I'm out of the loop enough to just go ahead and cut a release, so wanted to ask whether folks here are in a comfortable position to make a new release?

cc @guillaumeeb @lesteve

lesteve commented 9 months ago

@jacobtomlinson feel free to make a release if you have the time. Otherwise I can look into it, but this will probably not happen before beginning of March.

Just curious what is the unreleased functionality that they rely on?

jacobtomlinson commented 9 months ago

Is making a dask-jobqueue release documented anywhere? If I don't get to it I think beginning of March is a fine timescale.

They are using the worker_command kwarg in SLURMCluster to start GPU clusters that use dask-cuda.

lesteve commented 9 months ago

I think all we have for release doc is: https://github.com/dask/dask-jobqueue/blob/main/docs/release-procedure.md

OK so I guess this is the PR they want https://github.com/dask/dask-jobqueue/pull/606.

guillaumeeb commented 9 months ago

Hi there, no problem for a new release with the few changes made. I can try to do one this week or the next.

jacobtomlinson commented 9 months ago

Thanks @guillaumeeb

guillaumeeb commented 9 months ago

So, I have tagged a 0.8.3 version and pushed it on Pypi.

However, the conda forge feedstock pipeline failed: https://github.com/conda-forge/dask-jobqueue-feedstock/pull/20#issuecomment-1959220247.

I fear that this means even the generated pip package installation will always fail on Python >= 3.12.

So I think we need to update our versionneer (see #618), or even change to a more recent system.

I really have no experience into this, any advice @jacobtomlinson?

jacobtomlinson commented 9 months ago

We should merge #618 and make another release. The versioneer update is fine, I've done it in a few other places. I'm just pushing some extra commits to #618 to clean up versioneer even more and make pre-commit happy and then I'll merge it.

guillaumeeb commented 9 months ago

So do/can we override 0.8.3 release? Or should we do a 0.8.4?

lesteve commented 9 months ago

I think the simplest is to yank 8.0.3 on PyPI and do a 0.8.4 release.

You can not really override a PyPI release. There are something like wheel build tag see https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-name-convention that allow you to upload new files with the same version (it was used by scikit-learn 1.4 because there were issues). Another alternative is to do a 0.8.3.post1 release but I think it is simpler to do a 0.8.4 release.

jacobtomlinson commented 9 months ago

I think it is simpler to do a 0.8.4 release

Agreed

lesteve commented 9 months ago

FYI I have yanked the 0.8.3 on PyPI, I don't think there was an issue with it per se, since the problem happens only when you build the wheel on Python 3.12 IIUC. This does avoid a situation where 0.8.3 is on PyPI and not on conda-forge.

guillaumeeb commented 9 months ago

Thanks @jacobtomlinson @lesteve, I will relaunch the release process!

lesteve commented 9 months ago

Thanks, while I think of it, you probably want to modify the changelog to use 0.8.4 instead of 0.8.3 (i.e. not mention 0.8.3 at all in the changelog) in order to simplify things for people that read the changelog.

guillaumeeb commented 9 months ago

There seems to be issues with upload.pypi.org at the moment. I had some also this morning. Hope this will resolve in a few moment.

guillaumeeb commented 9 months ago

Okay, 0.8.4 is on Pypi, let's wait for conda bot.

guillaumeeb commented 9 months ago

Oh, crap, I didn't pull the upstream/main branch before tagging, I think I'll have to do over again. So sorry.

guillaumeeb commented 9 months ago

So I yanked 0.8.4, and uploaded a 0.8.5. Hope I didn't mess up this time.

jacobtomlinson commented 9 months ago

I've done that before! Thanks for getting this sorted.

lesteve commented 9 months ago

Same here, that happens to everyone, thanks for taking care of the release!

This is a good occasion to share this blog post by Brett Cannon: https://snarky.ca/what-to-do-when-you-botch-a-release-on-pypi/

guillaumeeb commented 9 months ago

There is still a problem in the conda-forge build: https://github.com/conda-forge/dask-jobqueue-feedstock/pull/22.

@jacobtomlinson do you know what should be done?

lesteve commented 9 months ago

Yeah I saw that, I think the quickest fix would be to remove the lsf_version function and the distutils import which is unused (was used at some point in the past).

So maybe a 0.8.6 release :sweat_smile:?

Maybe you want to make a conda-forge PR to check that it is green prior to making the PyPI release.

Alternatively, maybe simpler, adding a setuptools as a dependency in the conda-forge recipe would fix the conda-forge package.

guillaumeeb commented 9 months ago

Alternatively, maybe simpler, adding a setuptools as a dependency in the conda-forge recipe would fix the conda-forge package.

So I chose this solution for simplicity, I don't know if it is the cleaner, but now the release is on Conda too.

lesteve commented 9 months ago

OK great, let's close this one then!

GaetanLepage commented 6 months ago

Thanks for adding python 3.12 support. When is the next release (0.8.6) suppose to be available ?

lesteve commented 6 months ago

Unless I am mistaken, I believe dask-jobqueue 0.8.5 should have Python 3.12 support and has been released February 22 2024 https://pypi.org/project/dask-jobqueue/#history

tcztzy commented 4 months ago

I am waiting for solve this prolem #644

Unless I am mistaken, I believe dask-jobqueue 0.8.5 should have Python 3.12 support and has been released February 22 2024 https://pypi.org/project/dask-jobqueue/#history