Indicate in UI/API if a name has been prohibited

[ncoghlan]

(From the python-dev thread about the SK-CSIRT notifications)

Currently, if the PyPI admins reserve a name, it isn't straightforward for clients to identify that that is the case, as when a name is registered with no packages uploaded:

• the web UI reports a 404 response (so no registrant info is remotely available)
• the simple API reports a 400 response (simply indicating "registered, no uploads yet")

To make distributed PyPI security analysis easier, and to allow clients to alert users to potential security issues in their installation commands, I'm wondering if it may make sense to:

1. Track "reserved by admins" as a separate state for names in the DB model
2. Report 403 from both the web UI and the simple API for such packages

The idea behind this would be to be able to clearly distinguish reserved names that are only exposed to the known insider threat of compromise by PyPI admins, and names that remain open to use by anyone.

[dstufft]

This is effectively done now with https://github.com/pypa/warehouse/pull/2396. It isn't implemented exactly as laid out here, instead of having admins claim names we simply maintain a list of names which are blacklisted on PyPI.

This means that they show up as 404 in the Web UI and in the simple API (and they don't appear on the /simple/ index at all. From the outside they are exactly the same as an unregistered package (because well they are an unregistered package). The difference then comes when someone tries to upload a release for a project whose name has been blacklisted, they get a 403 The name 'foo' is not allowed., which is a distinct error message from any other errors on upload.

The total list of names that are blacklisted are not available to anyone but PyPI admins (and I don't think they should be, although it's a fairly small window by showing what packages we've blacklisted we provide the attacker a list of projects we feel are high value and what names they know won't work so they can focus on other names).

I'm going to go ahead and close this, but if you feel like the error message or status codes for blacklisted projects could be improved then I'm open to that.

[dstufft]

One note, Not all of the names that have been administratively claimed in the old way have been moved to the new blacklist. I moved over all of the names that I had moved onto my account over the years but @ewdurbin will need to do his own (and I'm not sure if Richard or MvL or anyone did the same in the past).

[ncoghlan]

Great to hear this is mostly implemented already! I think the one question worth considering is the response in the web UI and simple API for prohibited package names.

PEP 541 spells out a number of different reasons that use of a name may be prohibited. For folks attempting to register & upload a project using a prohibited name, it clearly makes sense to let them know that it isn't allowed.

However, I'm thinking it also makes sense to indicate it on download and web UI access, as that provides better information when previously published packages gets taken down as a result of the name being declared invalid.

Status quo: it just looks like the package got deleted somehow Proposal: affected users receive a clearly distinct "The name 'foo' has been prohibited by the server" error message

If the error message were to include some info on why it was prohibited, then affected users could also be informed as to whether or not their systems may have been exposed to malware.

This would make things slightly more efficient for attackers looking for names to compromise (since they'd get the 403 when checking to see if the package exists, rather than when attempting to upload the malicious payload), but I think the pay-off in gaining an inline notification channel to potentially compromised users would be worth it.

[dstufft]

I'm not entirely sure how I feel about that, but I've gone ahead and re-named this issue to reflect that scope and reopened it for discussion.

[ncoghlan]

Slight tweak to the title, since I prefer the allowed/prohibited terminology to whitelist/blackist.

[ncoghlan]

As per https://github.com/pypa/warehouse/pull/2396#discussion_r139297908, the other item to consider here is whether or not to expose some sort of predefined "reason code" for the name being prohibited (likely based on the categories listed in PEP 541).

That question applies to the existing 403 error reported for registration/upload attempts in addition to the potential 403 error for download attempts.