In pypi, it is impossible to reupload a removed file.

[Natim]

HTTPError: 400 Client Error: This filename has previously been used, you should use a different version.

[brechtm]

@merwok IIRC TestPyPI doesn't allow overwriting files either, so if you screw up you can't test with the same version number.

[brechtm]

+1 for @mikofski's suggestion for a sort of staging area on PyPI, similar to the functionality that devpi offers. This would allow you to upload a release, test it by supplying a different server to pip's --index-url option, and then "commit" the release.

Of course, it should be possible to drop the staged release in case you messed up.

But the question is, who will implement this? ...

[dstufft]

See pypa/warehouse#726.

[ncoghlan]

I'll also note that if anyone in this thread is using Python for professional purposes and would like to see additional development effort put towards PyPI, that the PSF accepts targeted contributions towards that end: https://donate.pypi.io/

It may also be worth contacting your Python support vendor or deployment platform provider, and asking them to invest time directly in upstream PyPI maintenance and development (since providing a Python runtime is only the first step in effectively supporting Python developers - vendors and platform providers should also be expected to provide reliable access to the wider Python ecosystem).

[atagar]

There we go. Took a handful of dry-run attempts but finally got a release I'm happy with. In case others want a safe approach for making releases here's what we're doing...

• Instructions
• Commit

Problems-between-chair-and-keyboard aside here's a couple things that accounted for some of the trial-and-error attempts...

• A multi-line 'description' in the setup.py successfully makes a package and uploads. PyPI however quietly crops the PKG-INFO, causing any data following it (like Categories) to be omitted.
• Evidently you can't set the bug tracker url through the setup.py.

C'est la vie. Now that we have a safe procedure for making releases I'm happy.

[merwok]

Two notes:

1) The provides field is not useful and not used, it would be best to remove it to avoid more confusion.

2) You can use setup.py upload for HTTPS uploads, it depends on your configuration in ~/.pypirc

[atagar]

Thanks! Dropped the provides clause. As for setup.py maybe PyPI could provide a more useful error than "Upload failed (403): Must access using HTTPS instead of HTTP"? If it gave a link to the upload instructions and that told people they can make a pypirc that would be useful. Unfortunately a quick search didn't have that.

[joernhees]

this is quite annoying... i just had a wheel for py2 pick up a build/src for py3 (that went through 2to3)... resulting ...-py2-none-any.whl is broken.

Considering that i noticed the problem within 2 minutes and immediately deleted the bogus .whl, it would be nice to have an option to pretend it just didn't happen. I see the arguments that releases should be immutable, but i'm quite sure that no one besides me pulled that file within the time-frame. As mistakes can and will happen, i'd love to not be robbed of a choice as a maintainer to decide what's best myself.

Why not at least have a grace period of a few minutes?

[dstufft]

2 minutes is plenty of time for hundreds of mirroring clients that run on a one minute cron job to find the file and pull it down immediately.

[joernhees]

mirroring clients as in pypi caches? they could and should update caches as well, no?

i have the feeling that pypi internal arguments are dictating maintainer experience here... why not give maintainers the information (like end-user download stats) to make well informed decisions and let them make their decisions?

at least however put a warning on that delete button on the pypi website as mentioned in https://github.com/pypa/packaging-problems/issues/74#issuecomment-260407394 ... it's a big UX hazard that trips up maintainers and is luring in the interface for at least 1.5 years now.

[joelmiller]

When you go to remove something, you get a warning message that it can't be undone. Could that warning be improved? Not only can it not be undone, but you will not be able to upload a replacement with the same name...

[atagar]

Yes, that's what's missing from the message. "Cannot be undone" means "cannot recover the deleted file" but it says nothing about it being impossible to re-upload another file.

Ideally the message would say something like "This cannot be undone and another file by the same name cannot be uploaded. This is a known limitation of PyPi and tracked on [link to issue]."

[joernhees]

all that's missing is someone hitting a button here: https://github.com/pypa/pypi-legacy/pull/591

[brainwane]

Some updates for folks in this discussion:

pypa/pypi-legacy#591 was merged in June 2017.

See https://github.com/pypa/packaging-problems/issues/75#issuecomment-347739479 for the ability to upload wheels for old releases, https://mail.python.org/pipermail/distutils-sig/2017-December/thread.html#31835 regarding immutability of metadata for past releases, and https://packaging.python.org/guides/using-testpypi/ for guidance on using the new Test PyPI.

Warehouse, the codebase behind the new PyPI, is available as a pre-production site at https://pypi.org, we're making steady progress on the developer roadmap thanks to funding from Mozilla's Open Source Support Program, and it's on its way to replacing the legacy PyPI site entirely within the next few months. We're currently seeking feedback from package maintainers about what does or doesn't work for you in the new interface, and in the next few weeks we'll be doing more outreach to the wider Python developer community. If you have tried the release deletion interface in Warehouse (including at https://test.pypi.org ) and the warning isn't strong enough, please let us know.

pypa/warehouse#726 is open to discuss how PyPI and Test PyPI can be better at providing staging/preview functionality for maintainers. #10 is open to discuss toolchain improvements for testing a package after creating it but before uploading it. There's an open issue pypa/twine#207 for adding a "dry run" option to Twine. (I am a new maintainer of Twine and just released Twine 1.10.0; please upgrade.) @atagar I saw your comment https://github.com/pypa/packaging-problems/issues/74#issuecomment-261802260 and followed the wiki history to check your code -- have you switched to Twine, and if so, does the approach you used still work?

Hope this helps.

[laike9m]

why?????? This is causing me so much trouble. :-1:

[im-n1]

Just ran into this. After reading almost whole thread I don't think there is something I can add. In one sentence - It's very stupid to have a test repository without the ability to do test stuff.

I hope the team will reconsider this. In the meantime we are going to spam your database with new and new "versions" because we can't rewrite the current one.

[sunbearc22]

I am using Test PyPI to try out my installation. I needed to make some changes to it but encountered errors. I finally tried deleting the entire package from Test PyPI, and made some changes to my script files, re-ran python3 setup.py sdist bdist_wheeland twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose. Still had issue:

Uploading distributions to https://test.pypi.org/legacy/
Uploading upgrade_pip_packages-0.0.1.dev0-py3-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.32k/7.32k [00:01<00:00, 5.25kB/s]
Content received from server:
<html>
 <head>
  <title>400 File already exists. See https://test.pypi.org/help/#file-name-reuse</title>
 </head>
 <body>
  <h1>400 File already exists. See https://test.pypi.org/help/#file-name-reuse</h1>
  The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>
File already exists. See https://test.pypi.org/help/#file-name-reuse

 </body>
</html>
HTTPError: 400 Client Error: File already exists. See https://test.pypi.org/help/#file-name-reuse for url: https://test.pypi.org/legacy/

I had not seen this discussion thread before doing the above. After reading it, my questions are:

1. Am I correct to conclude that with Test PyPI, after deleting my package, I can no longer reuse the same package name even though I am the owner of the package name?
2. I thought Test PyPI is ephemeral in nature. If so, can I reuse the deleted package name a day later?
3. According to #file-name-reuse, does the word "filename" means package name or something else?

[benoit-pierre]

You cannot reuse the same distribution name, .i.e. reuse the same source distribution or wheel file name (even if the whole package was deleted). You need to change the version number (or increase the build number for wheels).

[sunbearc22]

@benoit-pierre Thanks for clarifying. I hope PyPI developers can make the Test PyPI interface host packages ephemerally (i.e. temporarily) as it is for testing purpose. New user like myself will make mistakes and discover issues after packaging and needs to make fixes before actually uploading the package into the PyPI interface. I like to request the distribution name used in this test environment be non-permanent, reusable and not be linked to actual PyPI interface. My wish list.... Thank you.

[benoit-pierre]

You could use a release candidate version number: foo-1.0rc1, foo-1.0rc2, etc... Until all issues pushing to Test PyPi have been resolved and you release the final foo-1.0 to PyPI.

[1313e]

You could use a release candidate version number: foo-1.0rc1, foo-1.0rc2, etc... Until all issues pushing to Test PyPi have been resolved and you release the final foo-1.0 to PyPI.

Although I completely understand that, people often use the Test PyPI to "test" packaging, uploading and installing their packages. Which often involves making very minimal changes to the code and reuploading it again, tens and sometimes hundreds of times. It is incredibly annoying to have to remember yourself every time that you have to increase the version number after making a change. I did the same thing once with a package of mine, and I ended up with version 'v1.0.0rc127' on Test PyPI, which is a bit too much in my opinion.

Unlike normal PyPI, Test PyPI should allow for the same version number to be uploaded multiple times, overwriting the current version if it already exists (as long as you own the package name). As far as I know, Test PyPI is purely there for testing (hence the name) and therefore the policy that no distributions should ever be deleted (permanently) from the system, does not apply here. I hope somebody can take a good look at this, since having to increase the version number for every test release on Test PyPI does not add anything beneficial to anybody besides being forced to learn to increase the version number for every release (which is required for PyPI).

[joelmiller]

You could use a release candidate version number: foo-1.0rc1, foo-1.0rc2, etc... Until all issues pushing to Test PyPi have been resolved and you release the final foo-1.0 to PyPI.

Sure this can be done.

But the current warning does not advise people to take these steps.

A perfectly reasonable person can look at the instructions and expect that "this [deletion] cannot be undone" does not mean "once you've deleted this, you can't replace it with something else".

On Tue, Dec 4, 2018 at 4:20 PM Ellert van der Velden < notifications@github.com> wrote:

You could use a release candidate version number: foo-1.0rc1, foo-1.0rc2, etc... Until all issues pushing to Test PyPi have been resolved and you release the final foo-1.0 to PyPI.

Although I completely understand that, people often use the Test PyPI to "test" packaging, uploading and installing their packages. Which often involves making very minimal changes to the code and reuploading it again, tens and sometimes hundreds of times. It is incredibly annoying to have to remember yourself every time that you have to increase the version number after making a change. I did the same thing once with a package of mine, and I ended up with version 'v1.0.0rc127' on Test PyPI, which is a bit too much in my opinion.

Unlike normal PyPI, Test PyPI should allow for the same version number to be uploaded multiple times, overwriting the current version if it already exists (as long as you own the package name). As far as I know, Test PyPI is purely there for testing (hence the name) and therefore the policy that no distributions should ever be deleted (permanently) from the system, does not apply here. I hope somebody can take a good look at this, since having to increase the version number for every test release on Test PyPI does not add anything beneficial to anybody besides being forced to learn to increase the version number for every release (which is required for PyPI).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pypa/packaging-problems/issues/74#issuecomment-444310039, or mute the thread https://github.com/notifications/unsubscribe-auth/ACx3CepuUg9DajrgdCdytq0lWm_42j48ks5u1xExgaJpZM4F39Wz .

[pradyunsg]

pypa/warehouse#726 is open to discuss how PyPI and Test PyPI can be better at providing staging/preview functionality for maintainers.

[dstufft]

FTR, I don't think there is any opposition to Test PyPI not allowing re-uploads, just nobody has done the work to make that possible.

[bitmaks]

If you could delete a package and replace it with something different but with the same version, it can break their program

That's not what I am asking for.

I am asking for putting back the package I removed.

Exactly, I'm dealing with the same issue. I wasn't aware that this was a restriction imposed upon filenames and I had a large revamp of my code structure and I chose to remove my project only to try and add it back and find this error.

[torfsen]

FTR, I don't think there is any opposition to Test PyPI not allowing re-uploads, just nobody has done the work to make that possible.

@dstufft Could you give a brief description of what changes would be needed for this feature and how much effort it would be? This would help potential contributors (like me) to decide whether they can tackle it.

[alexlaurence]

You have to update your Github repo, then you can reupload. Just make a commit, and PyPi will allow the upload.

[OneAdder]

I think it still looks pretty weird. I just had a problem with this: I accidentally built my package the wrong way and uploaded it. It was not a problem within the package: I just forgot to delete the old build.

Now I had to change the version name for PyPI and explicitly say in the description that versions 0.8.5 and 0.8.5.post0 are the same version

[pradyunsg]

You don't have to say that - post releases are meant to be the same as the existing release and only differ in metadata.

[MartinThoma]

One important thing seems not to be mentioned so far: It is better for security and reliability to NOT allow changing the code of an uploaded version. As a user of a 3rd party library, I can once check it and then pin the version. My program is guaranteed to work the same way. Even if the account of the maintainers get hacked, the worst thing that can happen is that the package is deleted. So it either works the same way or not at all. But at least the attacker cannot bring malicious code in another code base.

[Natim]

Note that if you remove the project altogether and create it again, you can reupload code from previous version.

[1313e]

One important thing seems not to be mentioned so far: It is better for security and reliability to NOT allow changing the code of an uploaded version. As a user of a 3rd party library, I can once check it and then pin the version. My program is guaranteed to work the same way. Even if the account of the maintainers get hacked, the worst thing that can happen is that the package is deleted. So it either works the same way or not at all. But at least the attacker cannot bring malicious code in another code base.

Good point. In that case, you could be allowed to delete and reupload a release if that happens within (let's say) a week after first release. That way, we can replace releases when we made a mistake, while the problem you mentioned will not be an issue.

[brianmay]

Note that this has limitations. e.g. AFAIK I could upload a source file, and sometime later a malicious actor could upload a wheel file for the same version that has malicious content (i.e. it doesn't have to match the source).

[im-n1]

IMHO pypi.org should be immutable. test.pypi.org should be mutable because it's test environment.

[joelmiller]

I don't think many of the commenters have been suggesting that pypi itself should allow resubmissions with the same name, and I agree it should be immutable. (but the error messages shouldn't say "deleting can't be undone" if in fact "deleted files can never be replaced"). There's a lot of problems there - different mirrors might have different versions, malicious actors, etc.

I'm okay with different mirrors of a test environment having different versions, and a malicious actor can't really get far by infecting something that's explicitly a test.

[ikamensh]

It is essential to disallow delete and upload of existing file. Without this you can't guarantee anything; malicious actor could inject code to your project and you wouldn't be able to do anything about it.

[brianmay]

@ikamensh It isn't quite that simple. I believe there is nothing stopping an upload of the same version of the package in another format, e.g. if you only upload a source file, a malicious wheel file can be uploaded at a later date, which the clients will use in preference. So I am not entirely convinced this is a security gain. At least it does stop replacing already uploaded files however.

Strategies such as pipenv which store the checksum of the file are probably are the better approach to ensure the package doesn't change unexpectedly.

[brainwane]

I would welcome thoughts from @mnm678 and from @xmunoz on this.

[mnm678]

I agree with the general sentiment of this thread: once a package is uploaded to PyPI, it should not change. If an attacker was able to compromise a project, they should not be able to replace an existing package with arbitrary code. Not only that, but even a well intentioned change to an existing package version could break a client's code. If a client pins a version of a package, they should get the same code every time that package is downloaded.

Earlier in the thread, @dstufft suggested a soft-delete function which would allow developers to undo the deletion of a package without reuploading the file. This seems like a good compromise between the security risks of new uploads and the usability problem of accidental deletion. In this case an attacker could undo the deletion of a version, but would not be able to add malicious code to that version (although they could still add a new, bad version unless something like PEP 480 is used to verify the developer's identity).

However, the discussion about PyPI-test is a different story. The security risks in a test repository are less important as any packages are not used in production, so I don't see a problem with allowing test versions of packages to be replaced.

[uranusjr]

FYI the soft-deletion feature has been formalised as PEP 592 “yank.” It is not yet implemented for PyPI (pypa/warehouse#5837).

[brainwane]

Relatedly: an implementation proposal for a "draft releases" feature on PyPI https://github.com/pypa/warehouse/issues/726 is now being discussed at https://discuss.python.org/t/feature-proposal-for-pypi-draft-releases/3903 . Check it out to see whether you have comments to share there. @alanbato would like comments on the proposal by 30 April (10 days from now) so he can go ahead and start implementation work.

(More context on distutils-sig.)

[brainwane]

Now that PEP 592 is accepted and implemented pypa/warehouse#5837, I hope people in this thread will consider looking at the yanking feature and see which of your use cases it suits.

[TylerGubala]

With the new policy you basically say: You've ONE SINGLE TRY and that one SHOULD WORK. No chance for a 2nd try. IMHO this isn't the purpose of a testing system and breaks the whole "we've a testing repo" idea. To be honest, I think this only leads to annoyed developers and a lot of "crippled versions" because developers couldn't properly test their versions before going live.

Literally this. In my repo, Blenderpy I have the additional headache that I often don't know exactly everything to test in advance, as there can be a lot of edge cases in the precompiled C/C++ code that may be remedied by configuring with different cmake arguments.

Why cannot you do package x.y.z.dev0 and then package x.y.z.dev1?

For example, in my case people performing pip install bpy==2.81 want the api version 2.81.

So now I effectively have my own version squatting on me, wherein people factually cannot install the proper version because I made an honest mistake.

So I think there are a lot of important considerations here. There is the "version contract", but then there are case by case times where I think that does not apply. I should think that if I accidentally upload some bad version of my own repo, then go back and try to remedy the issue, that does not critically break the version contract.

I'm sorry but I do not understand the concept of shackling yourself to a "version contract" that basically screws yourself and your users in the end.

[arigo]

I was just told about "build numbers", which apparently let us upload several wheels for the same version number and platform, just by giving them slightly different names:

wheel-0.31.0-py2.py3-none-any.whl wheel-0.31.0-1-py2.py3-none-any.whl wheel-0.31.0-2-py2.py3-none-any.whl

It would seem to be exactly the feature that lets us re-upload files---the same file, or one with a fix. It would have saved the OP and myself the pain of making completely new versions just to fix one obscure platform wheel---including, lately, breaking another, widely-used platforms while doing so.

Why is this not the solution? Why is the "build number" feature not widely known?

[duck-rh]

I did not see any progress on #10 and related pypa/warehouse#726, so is there any hope to be able to reupload on testpypi so we can check the result before the final upload on pypi? I'm fine with another solution but it seems not much progress was made since such kind of discussions started in 2013.

[duck-rh]

@arigo I tried your trick to rename the files (thanks rename) in order to avoid changing the sources and ended-up with this error: "Only one sdist may be uploaded per release". Since I cannot exclude it to be sure my fixes are properly taken into account, I tried deleting the version and got the same error as this issue; fortunately bumping the "build number" again worked.

Seems the best workaround so far though.

[makseq]

@arigo Unfortunately I have a problem with this approach:

pip install label-studio==0.9.0
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
Collecting label-studio==0.9.0
  Using cached label_studio-0.9.0-2-py3-none-any.whl (18.9 MB)
ERROR: Requested label-studio==0.9.0 from https://files.pythonhosted.org/packages/7f/32/da8e6b279134e96c8b835bba3c20b8638473fed4f514b10472582258ae8b/label_studio-0.9.0-2-py3-none-any.whl#sha256=1363a13887d6351b720c89c43be78d41d42fff1cee56cebdd4fbf60077668fb9 has different version in metadata: '0.9.0.post2'

[Tasty213]

Hi, I just removed a package version because my internet connection failed midway through and so it thought I had uploaded a file when I actually hadn't. I didn't realise this would mean I could never use that version number again because there's no warning. I think this is a bit absurd as now I'm going to have to make a new release (v0.1.4) without making any changes.

There should be a warning before you delete versions and files saying that you can never use those names again to ensure users understand what they're doing.

[henryiii]

You can upload a post version, v0.1.3.post1 - this will be treated like the original, --upgrade will know they are the same, etc. You can also upload missing files without starting a new version if one or more wheels didn't upload (or the SDist).

[microprediction]

I just bumped the version number a few times, tried again and it worked fine, and got on with my day.