search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.61k stars 2.82k forks source link

Package naming scheme changed to use underscores without prior notice #35204

Closed glaubitz closed 1 week ago

glaubitz commented 6 months ago

While updating individual component packages of the Python Azure SDK on openSUSE Tumbleweed today, I noticed that the package naming scheme has changed to use underscores instead of hyphens (s/-/_/g):

refs/tags/azure_mgmt_largeinstance_1.0.0b1 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_frontdoor_1.2.0 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_cdn_13.1.0 Mon Apr 15 15:43:03 2024 +0800

I'm not sure whether this accidental or intentional, but in any case it's a breaking change for downstream distributions which have to adjust source archive names now.

Can this be reverted or or this there a valid reason to use underscores now (which are rather uncommon on PyPi in my experience)?

weshaggard commented 6 months ago

It looks like there was a tooling change to the setuptools package that caused this change.

@msyyc it seems like you by-passed the failed CI in the following PRs (https://github.com/Azure/azure-sdk-for-python/pull/35200, https://github.com/Azure/azure-sdk-for-python/pull/34980, https://github.com/Azure/azure-sdk-for-python/pull/35201) and then released them with those failures which causes for incorrect release tags to be created. We should get those release tags fixed and we might even need to yank the packages if they are incorrect as well.

msyyc commented 6 months ago

Hi @weshaggard This is first time we meet these failures which are treated as fake warning by wrong. To meet the expected SDK release date, we bypass the CI check but now it seems "correct" error. Since we don't change anything about release pipeline, I don't understand why these failures happen. Appreciate if you could provide guidance how to fix it.

lmazuel commented 6 months ago

While updating individual component packages of the Python Azure SDK on openSUSE Tumbleweed today, I noticed that the package naming scheme has changed to use underscores instead of hyphens (s/-/_/g):

refs/tags/azure_mgmt_largeinstance_1.0.0b1 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_frontdoor_1.2.0 Mon Apr 15 15:43:03 2024 +0800
refs/tags/azure_mgmt_cdn_13.1.0 Mon Apr 15 15:43:03 2024 +0800

I'm not sure whether this accidental or intentional, but in any case it's a breaking change for downstream distributions which have to adjust source archive names now.

Can this be reverted or or this there a valid reason to use underscores now (which are rather uncommon on PyPi in my experience)?

@glaubitz see https://setuptools.pypa.io/en/stable/history.html#v69-3-0 @xiangyan99 is currently fixing it by pinning to an old setuptools, and we will discuss a plan to move forward with newer version (I don't know yet how we will approach it). Let me know if you have a preference, and we can factor that in our discussion.

weshaggard commented 6 months ago

To help with some of this I've pushed the following tags:

https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-largeinstance_1.0.0b1 https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-frontdoor_1.2.0 https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-cdn_13.1.0

I don't know if the packages that are published are going to be impacted or not but we at least need to have the tags fixed otherwise other systems start breaking. For now @xiangyan99 pinned the setuptools with PR https://github.com/Azure/azure-sdk-for-python/pull/35212 but once @scbedd is back we probably need to spend more time investigating options.

scbedd commented 6 months ago

Relevant discussion in setuptools repo

It's definitely an intentional change on setuptools part. We should also update our package discovery to handle the new canonicalized names. That's not the part that actually concerns me though.

I'm currently digging into why our pinned setuptools version is getting overridden by the new setuptools version. Will close this issue after I understand the origin/update our wheel discovery logic to handle the new setuptools.

scbedd commented 6 months ago

Updates to the base image versions caused the installation of some global packages to not be updatable by a standard pip install. Even with force enabled.

Merged #35343 to resolve the weird pinning situation.

glaubitz commented 4 months ago

This issue has returned with azure-mgmt-appplatform and azure-mgmt-oracledatabase using underscores in their names while the latest version of azure-ai-ml still using dashes.

The behavior seems inconsistent now.

weshaggard commented 4 months ago

@scbedd could you please take a look https://github.com/Azure/azure-sdk-for-python/releases/tag/azure_mgmt_oracledatabase_1.0.0

scbedd commented 4 months ago

@scbedd could you please take a look https://github.com/Azure/azure-sdk-for-python/releases/tag/azure_mgmt_oracledatabase_1.0.0

  1. Yes. This is due to the upgraded setuptools that I added this past Friday.
  2. Will patch our builds to produce the correct file names, regardless of the setuptools normalization practices for now
glaubitz commented 4 months ago

Thanks for the fix! Could you retag these affected packages afterwards?

scbedd commented 4 months ago

Thanks for the fix! Could you retag these affected packages afterwards?

  • azure_mgmt_edgezones
  • azure_mgmt_oracledatabase
  • azure_mgmt_appplatform

Absolutely. Apologies for inconvenience.

glaubitz commented 4 months ago

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

scbedd commented 4 months ago

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

The packages have been retagged, and the relevant releases updated to reflect the same.

glaubitz commented 4 months ago

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

The packages have been retagged, and the relevant releases updated to reflect the same.

Would it be possible to push the packages onto PyPi again? These can still be downloaded with the old filenames only.

scbedd commented 4 months ago

No worries. I'm withholding the updates to these packages in openSUSE for the time being.

The packages have been retagged, and the relevant releases updated to reflect the same.

Would it be possible to push the packages onto PyPi again? These can still be downloaded with the old filenames only.

If you need the file names updated, I'd likely just cut a patch release for all 3. I can add additional wheel entries for a version, but can't republish the sdist to my knowledge. I'll double check that though.

EDIT: Yeah can't publish sdist twice.

glaubitz commented 4 months ago

If you need the file names updated, I'd likely just cut a patch release for all 3. I can add additional wheel entries for a version, but can't republish the sdist to my knowledge. I'll double check that though.

I suggest another release maybe with a "post1" suffix or so.

We're pulling the source tarballs semi-automatically from PyPi for updating the RPM packages of the SDK.

scbedd commented 3 months ago

@glaubitz sorry for delay hit some unexpected foibles.

I re-released with the post1 versions.

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

glaubitz commented 3 months ago

@glaubitz sorry for delay hit some unexpected foibles.

I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions.

Will update these packages tomorrow as it's already late now here across the lake.

scbedd commented 3 months ago

@glaubitz sorry for delay hit some unexpected foibles. I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions.

Will update these packages tomorrow as it's already late now here across the lake.

No problems! Just LMK if you hit any issues. I don't expect you will but I'll jump on em if you do.

glaubitz commented 3 months ago

@glaubitz sorry for delay hit some unexpected foibles. I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions. Will update these packages tomorrow as it's already late now here across the lake.

No problems! Just LMK if you hit any issues. I don't expect you will but I'll jump on em if you do.

I just had a go and I'm afraid to say the issue is not fully fixed.

While the tarball name has been corrected, the source folder name is still using underscores.

scbedd commented 3 months ago

@glaubitz sorry for delay hit some unexpected foibles. I re-released with the post1 versions.

* [azure-mgmt-edgezones](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-edgezones_1.0.0b1.post1)

* [azure-mgmt-appplatform](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-appplatform_9.1.0.post1)

* [azure-mgmt-oracledatabase](https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-oracledatabase_1.0.0.post1)

Famous last words we shouldn't see this again, as we are doing some intervention to assure package is format we expect. setuptools updates shouldn't cause any problems going forward.

Great, thanks a lot! I highly appreciate the effort by the Azure SDK team to ease packaging the Azure SDK for downstream distributions. Will update these packages tomorrow as it's already late now here across the lake.

No problems! Just LMK if you hit any issues. I don't expect you will but I'll jump on em if you do.

I just had a go and I'm afraid to say the issue is not fully fixed.

While the tarball name has been corrected, the source folder name is still using underscores.

Yeah I did just update the tarball. I didn't really want to go backwards on setuptools.

You mind pointing me at the automation? I'll help with a PR if you don't mind. Unfortunately part of the reason we hit this issue was due to a compatibility problem with older setuptools and the new agent images. Frankly I probably should have followed up with you to coordinate updating the automation for newer setuptools when this first occured. It slipped my radar unfortunately.

Re-opened till real resolution.

scbedd commented 3 months ago

Well, I got earlier an earlier version of setuptools working in our pipelines. Any further releases off of main should use setuptools==69.2.0 as well.

@glaubitz Not urgent. Whenever you get a chance, please try post2 versions:

Now that main is properly unblocked, I definitely want to help provide a patch for your automation. I don't want this to happen a 3rd time!

EDIT: leaving this issue open until confirmation of resolution this time.

glaubitz commented 3 months ago

Looks like another underscore-named tag just slipped in:

refs/tags/azure_search_documents_11.5.1 Tue Jul 30 10:45:28 2024 -0700
scbedd commented 3 months ago

Looks like another underscore-named tag just slipped in:

refs/tags/azure_search_documents_11.5.1 Tue Jul 30 10:45:28 2024 -0700

This was bad timing. The release branch forked off of main while the incompatible bits were present, so the released version included the underscores 👎

Can you please point me to your repo where your automation parses this value? I'd like to provide a patch if possible. While main will be fine for you until we intentionally upgrade setuptools, I can't guarantee that other patch releases don't ship and break you.

scbedd commented 2 months ago

Hey @glaubitz ? Unfortunately I'm now getting hit by CG alerts that are going to force me to upgrade past setuptools>70.

This is going to break you again, and while I really don't want to do that, I don't really have a choice given the mandates I'm under.

I've got some time to defer the CG alert, but I'd like to respond in the next couple of days. Can I please assist you with a patch so I don't break you when that occurs?

glaubitz commented 2 months ago

Can I please assist you with a patch so I don't break you when that occurs?

Sorry, I was on vacation. What do you suggest? I just noticed that azure-template upstream tarball name was renamed to azure_template.

scbedd commented 2 months ago

Can I please assist you with a patch so I don't break you when that occurs?

Sorry, I was on vacation. What do you suggest? I just noticed that azure-template upstream tarball name was renamed to azure_template.

Yeah, that version of azure-template you're seeing is a test release from this PR #37035 addressing some component governance alerts related to setuptools. This is a sdist normalization change that came with setuptools>=69.2.3. The tag will maintain our existing formatting, but the source distribution will begin to match the whl naming conventions.

I don't know where in your process you rationalize this, but I hope it will be as simple as

replace - in tag with _ to get expected tarball name.

If you point me to a place in your process I'm totally willing to help with that!

I will delay merge of #37035 a couple more days while we figure this out!

glaubitz commented 2 months ago

Can I please assist you with a patch so I don't break you when that occurs?

Sorry, I was on vacation. What do you suggest? I just noticed that azure-template upstream tarball name was renamed to azure_template.

Yeah, that version of azure-template you're seeing is a test release from this PR #37035 addressing some component governance alerts related to setuptools. This is a sdist normalization change that came with setuptools>=69.2.3. The tag will maintain our existing formatting, but the source distribution will begin to match the whl naming conventions.

I don't know where in your process you rationalize this, but I hope it will be as simple as

replace - in tag with _ to get expected tarball name.

If you point me to a place in your process I'm totally willing to help with that!

I will delay merge of #37035 a couple more days while we figure this out!

OK, so if I understand you correctly now you consider the change from hyphen to underscore inevitable.

If that's the case, we'll just have to accept it and I have to adjust the upstream source names for future packages.

scbedd commented 2 months ago

OK, so if I understand you correctly now you consider the change from hyphen to underscore inevitable.

I do. this is the specific setuptools version where this name transition happens

With respect to #37035, I am going to be merging it today. I apologize for the disruption to your process that it will cause, but I need to get these alerts off my plate.

My offer for assistance remains open!

xiangyan99 commented 1 week ago

We have merged the change,

Closing the issue and please feel free to re-open it if it is not solved.