[Feature]: NuGet tooling to view package prefix reservation

[nkolev92]

NuGet Product(s) Involved

NuGet.exe, Visual Studio Package Management UI, Visual Studio Package Manager Console, MSBuild.exe, dotnet.exe

The Elevator Pitch

With package source mapping, you can establish a 1-N relationship between a package identity and a source. When using package prefix patterns, you may want to consider things like, is this prefix reserved on nuget.org, and use that as a means to make a decision.

For example, one would want to know given a package id, where it's prefix is reserved, or if so, which one.

Note that there might not be easy ways to get this data from the servers, so this might span client + server.

Additional Context and Details

No response

[JonDouglas]

This is a cool idea. It almost seems like a mini security feature.

[joelverhagen]

This would require a protocol enhancement and careful consideration of how much information we want to share. It's not immediately clear to me that there should be an easily accessible list of all namespaces in existence.

The only information available today is whether a specific package ID is reserved. This may be due to a prefix reservation or due to an exact ID reservation. This aspect is opaque to the package consumer.

edit: I like the idea too 😃

[nkolev92]

This would require a protocol enhancement and careful consideration of how much information we want to share. It's not immediately clear to me that there should be an easily accessible list of all namespaces in existence.

I think at this point we just want to listen/learn how people use prefix reservation client side and how they'd use it in the context of source mapping in particular.

[nkolev92]

Triage: Hey @JonDouglas can you please create a respective issue server side?

[glopesdev]

The only information available today is whether a specific package ID is reserved.

@joelverhagen can you elaborate on this? Is there indeed a way to check whether a specific package ID is reserved without actually trying to push a package?

In general it seems to me that a bit more transparency is required from nuget.org regarding package prefix reservation. Just recently I have been blocked from uploading a package falling under a prefix I have used in the last 10 years and when contacting NuGet support the only reply I received was the prefix had been reserved by NuGet for "security reasons".

No one actually seems to hold the prefix since none of the packages who might fall under the prefix rule have the verification checkmark, and no other explanation was given.

Package ID prefix squatting is already a problem, but when this is done by the server itself with no transparency or accountability this raises a whole other level of community trust, essentially undermining the confidence of package authors that nuget.org is fit to act as a custodian for community packages.

[joelverhagen]

Thanks for the great feedback @glopesdev! Let's talk about two separate issues:

First.

No one actually seems to hold the prefix since none of the packages who might fall under the prefix rule have the verification checkmark, and no other explanation was given.

For your specific prefix or package push being blocked, could you send an email to account@nuget.org about this? We work directly with package owners to give access to ID prefix reservations and evaluate a variety of factors to determine whether the reservation should be assigned. If you've received a response in the past that blocked you and prevented you from gaining access to a namespace that you want, I'd like to revisit that over email and figure out what we can do.

Second.

Generally, I agree adding visibility to namespace reservation would be desirable. However, there are several edge cases that we need to be careful about. For example, consider a case where an organization has requested to protect a space of package names that would clearly refer to their organization (e.g. {SpecificCompanyName}.*) but they have not yet pushed any packages. Should this fact be easily discoverable? Should someone be able to enumerate all of such cases and find gaps so they can pursue their own name squatting efforts in a more targeted manner? I'm not sure! Sure, folks can "spear fish" and push test packages with a variety of patterns and prefixes to manually discover the existing reservations, but such a high scale pattern would be noticed and even blocked by our system.

I'll speak with @clairernovotny about this point. She's the PM owner of ID prefix reservation so her perspective is essential.

Package ID prefix squatting is already a problem, but when this is done by the server itself with no transparency or accountability this raises a whole other level of community trust, essentially undermining the confidence of package authors that nuget.org is fit to act as a custodian for community packages.

Fundamentally, ID prefix reservation is a feature meant to improve clarity and trustworthiness in the .NET package ecosystem. If it's having the opposite effect, that's definitely something worth discussing!

I can confidently say that NuGet.org admins are not prefix squatting, at least in the traditional sense I am aware of (where there is a notion of extortion, malicious intent, or antagonism). We do not intend to permanently block legitimate package authors from pushing packages to namespaces that they have a good claim to.

If a namespace is reserved and actively used by party X and party Y wants access too, we would generally follow the Dispute Resolution flow. Party X has a strong say in how that namespace gets used.

If the namespace is reserved and is not being actively used or no longer has an owner (it sounds like you're more talking about this case), then we would evaluate the new requestor's claim similar to how we would evaluate a request for a completely new, unclaimed namespace. This application process is described in our documentation.

Finally, if there's a specific direction you'd like to go with improving ID prefix reservation visibility, please consider using the NuGet Proposal Process to describe how you think this new feature would work.

[glopesdev]

@joelverhagen Thank you so much for the care and attention you put in this response, it really goes a long way. I can only imagine how many stakeholders a large scale service like nuget.org has to balance. It is easy to start feeling lost in the automation and processes, and it is somehow always refreshing to hear that humans are still listening.

I'm happy to discuss over email, but I would like to try and elaborate on some more general feedback below.

My email exchanges have so far been with support@nuget.org as this was the address suggested by the package push error message:

Response status code does not indicate success: 409 (The package ID is reserved. You can upload your package with a different package ID. Reach out to support@nuget.org if you have questions.).

I sent a relatively detailed message explaining the whole situation, and got the following reply back:

Hi,

Thanks for contacting NuGet support. The ID prefix is reserved by us for security purposes. Please follow instructions here if you would like to reserve this namespace:

ID Prefix Reservation | Microsoft Docs

Thanks, NuGet Admin.

Given this, I wanted to offer the following points as feedback to hopefully improve future interactions:

1. The error message could perhaps be updated to include the link for the ID Prefix Reservation page directly, as it might save one round of interactions with support (they did end up referring to the link ultimately).

2. ID package prefix reservation as implemented currently makes batch uploading of multiple packages with cross-dependencies dangerous. Because package prefix only applies to new packages, and not old packages, some packages might be pushed successfully, while others are rejected. However, if the successfully pushed packages declared dependencies to the rejected packages, you effectively end up with a broken package that is impossible to fix quickly. This problem alone I think would already merit a new flag for an "hypothetical push", a kind of quick check to a NuGet server to check whether pushing a package with the specified ID would succeed or not, without actually pushing the package. At least this way one would not be caught by surprise at the very last step, and would be possible to plan a follow-up move without having to immediately deal with the "fire" of a broken package consumed in bulk by the community.

3. I completely understand your point about the downsides of publishing a registrar with all the prefix reservations, especially those made ahead of time. However, the reply from support stating "The ID prefix is reserved by us for security purposes." is erring in the opposite direction, being too opaque and potentially misleading given your explanation. Taking by us at face value, it is reasonable to infer that the block is in fact being imposed by NuGet itself, and not by some other party. But it was the appeal to security purposes without further qualification or explanation that I found most frustrating and worrisome, and prompted the comments above about lack of transparency. The reply as it is provides no clear feedback about what criteria were actually used to reserve the prefix, and whether this might happen again with other prefixes. If indeed the goal of this communication is to leave ambiguous who is making the ID prefix reservation, then I feel there must be some other way of clarifying or communicating the reason without leaking sensitive information. At least when someone else has reserved a package and you can see them, you can try to reach out to them for an exemption, or simply to make your case.

4. The ID prefix reservation and verification process does make sense for large-scale publishers, but I feel it comes across as too daunting and overwhelming for individual developers who might be volunteering their time for many years to maintain a high-quality package, especially if the initial prompt is an opaque reply such as the one given in thee email above. I honestly thought I had no chance going through this route given that message. The need to invest the extra resources might just be too big of a burden to bear, which tilts the balance of power towards organizations and away from individuals, especially without more tools for transparency to help make planning decisions.

Hope this helps clarify and frame my earlier feedback and again thank you for your reply. I will consider elaborating these proposals further in some of the links you sent out.

[joelverhagen]

I'm happy to discuss over email, but I would like to try and elaborate on some more general feedback below.

Yes, I agree with your split. I also prefer to having much of this discussion in the public since others can benefit. To be clear, my team typically aligns amongst ourselves using internal communication mechanisms so we can provide a consistent and clear message externally, so you might see me or others on the NuGet team leap ahead in the conversation after we've met internally.

I have found your support request so we can work on your specific case over email. We may need to tune our support processes in addition to the product/policy suggestions you have here.

The error message could perhaps be updated to include the link for the ID Prefix Reservation page directly, as it might save one round of interactions with support (they did end up referring to the link ultimately).

Yes, great idea. I think we could set up an aka.ms shortlink that can easily fit in a CLI error message. Additionally, putting support@nuget.org is wrong. account@nuget.org should really be mentioned instead, if any email address is mentioned at all since that's the mailbox we use for ID prefix reservations. Don't look at the git blame on who implemented that message 😄...

I've tracked your suggestion here: https://github.com/NuGet/NuGetGallery/issues/9075

ID package prefix reservation as implemented currently makes batch uploading of multiple packages with cross-dependencies dangerous.

Yes, agreed. We have a long-standing batch upload issue (https://github.com/NuGet/NuGetGallery/issues/3931) that I just linked to another user on Twitter in the past day or two. This problem generalizes to more than just ID prefix reservation issues. Any sort of problem during upload and validation can cause a break in the package graph. We have had this problem impact both Microsoft teams and members of the OSS community. Please comment on that issue with your specific scenario and upvote it. We use upvotes as a tool to prioritize work.

If indeed the goal of this communication is to leave ambiguous who is making the ID prefix reservation, then I feel there must be some other way of clarifying or communicating the reason without leaking sensitive information.

I think you're hitting on the core issue: we as a team have situations where we need to be terse/minimal in the level of detail we share for security and privacy reasons. That's part of our job as stewards of the .NET package ecosystem. It's a balance between oversharing (leading to more problems) and undersharing (leading to loss of trust or unnecessary frustration, as we have here). I'd like to keep this conversation scoped to ID prefix reservations so I'll leave it at that.

I need to talk to the team about how much we can share here. At this moment, I don't have any specific guidance beyond this:

If your package upload is blocked due ID prefix reservation, and there is no clear package owner you can contact about getting access, please contact account@nuget.org and provide details as to why you should believe you should get access to the namespace. Follow the steps at ID prefix reservation application process to the best of your ability to expedite the process and ensure all the required details are there.

In your specific case, @glopesdev, no further action is necessary. I'll look at the email you already sent in and make sure we've done everything we can.

I honestly thought I had no chance going through this route given that message. The need to invest the extra resources might just be too big of a burden to bear, which tilts the balance of power towards organizations and away from individuals, especially without more tools for transparency to help make planning decisions.

This is phenomenal feedback. Personally, I try to be sensitive to the cases where hobbyists, passion project owners, or individual OSS contributors are left at a disadvantage. I have not thought deeply about ID prefix reservation with this lens until now so I appreciate your persistence. I want .NET to be a vibrant and welcoming place for both individuals and organizations.

@clairernovotny, any ideas on this subject?

[clairernovotny]

Hi @glopesdev, thank you for that additional insight.

I agree that there's a potential issue when uploading a set of packages. I'll work with the team to see how we might be able to make improvements to the overall process.

[glopesdev]

@JonDouglas @joelverhagen

Once again I find my work blocked by silent package prefix reservation, this time directly crippling the entire development community of my most important open-source project. I already sent emails to all relevant contacts as I spent the last couple of days trying to come to terms with the implications of what just happened. However, more crucially I have been trying to understand what exactly is wrong with the process that NuGet is currently following with package prefix reservation that is allowing this to happen multiple times, and I wanted to leave here a public record of those reflections.

I care deeply for the NuGet community, so I really wanted to give constructive feedback and find the root of the problem, and I find myself coming back to this thread, our discussion above, and specifically to this statement:

Generally, I agree adding visibility to namespace reservation would be desirable. However, there are several edge cases that we need to be careful about. For example, consider a case where an organization has requested to protect a space of package names that would clearly refer to their organization (e.g. {SpecificCompanyName}.*) but they have not yet pushed any packages. Should this fact be easily discoverable?

Is this the driving motivation for enforcing package prefix reservations without any notification to existing prefix users? Especially without even requiring the entity that obtains the reservation to upload any package that effectively uses the prefix? If this is the case, then I believe this is the root of the problem, as it effectively breaks the social contract by which the NuGet community is intended to operate, as stated by the NuGet Project Governance, in at least two crucial aspects:

"Users are community members who have a need for and use NuGet, as package consumers and/or authors. Users are the most important members of the community: without them, the project would have no purpose. Anyone can be a user; there are no specific requirements".

• This statement is directly contradicted by the fact that entities can reserve package prefixes which directly damage the activities of users, without being themselves users of the community (since they can reserve a prefix without ever even contributing a package in the first place).

"Building community trust in the governance of an open-source project is vital to its success. To that end, decision making must be done in a transparent, open fashion. Discussion about the project’s direction must be done publicly. The community should never be caught off-guard by a decision by the Benevolent Dictator."

• Twice our community was blocked from uploading packages by silent package prefix reservation without there ever being any communication or warning that a prefix was now under reservation. Developers were simply barred overnight from pushing packages to NuGet, in the middle of their work activities, by a decision that came from nowhere.

The violation of these core tenets directly damages trust in the NuGet "benevolent dictatorship" model, since NuGet is approving and issuing crippling enforcement decisions without any notification to potentially aggrieved parties.

From a purely technical perspective with package prefix reservation, this is even more crippling since NuGet strongly recommends package IDs to match assembly names, which makes sense for dependency resolution mechanisms, but makes this kind of enforcement devastating for growing projects spanning multiple packages, since one cannot simply change the name of an assembly without a major breaking change.

I suggest below a few basic principles that I think should be applied whenever package prefix reservation is considered:

• If there is prior use of the prefix, and especially of the form {rootNamespace}.{assemblyName}, notify the current users of the prefix that such an application to reserve the prefix is being made. If you don't want to create a public record that is fine, but at least let possibly affected projects know ahead of enforcement.
• Allow for an opposition "grace period" ahead of enforcement where existing users can voice their concerns, especially if the package prefix reservation application is intending not to be "public".
• Take these concerns into consideration ahead of enforcement, ideally when evaluating the application.
• Require the reserving party to use the package prefix within a limited time window following reservation. If there are legal reasons in place requiring enforcement without package publication, then these should be made public (basic principle of sound law enforcement is to provide notice of accusation and the right to a fair trial before passing judgment and sentence).

I think there needs to be a deep discussion about what kind of register NuGet wants to become. The current one with strict and opaque package prefix enforcement is a major shift from the kind of respectful community that NuGet fostered and where I grew up in as an open-source developer. If the NuGet Project Governance document is no longer relevant, this fact should be clearly communicated to the community so expectations can be managed ahead of time. I currently fear for several open-source projects that have not done their "due diligence" of reserving a package prefix, meaning they are effectively at risk of being shutdown by arbitrary 3rd parties who do not care for community engagement.

[JonDouglas]

I agree with your response in spirit. It deeply troubled you and therefore it could be vastly improved. As prefix reservations have started to become more popular in the last couple years alone, it has shown difficulty of scale.

Take the pypi process for example. Someone opens an issue on GitHub and gets the request granted. This also acts as a public ledger of some sort. https://peps.python.org/pep-0541/ and https://github.com/pypi/support/issues?q=is%3Aissue+is%3Aopen+namespace+label%3A%22PEP+541%22

Where the gap today lays I believe is in the "knowing" through tooling or some means. We don't really do this well outside of seeing a UI affordance if it is already granted and any current support emails of someone's pending ask. It should be more obvious and self-serve. EX: If you try a prefix on a form and it's already reserved on nuget.org, it should show some type of validation stating it's already reserved by X org or X owner, maybe the ownership history, and perhaps instructions for disputing if you own the rights(DMCA/copyright claim). Otherwise it should allow you to submit a request that will be pending approval for the next appropriate business day. This I think would really help the spirit of this issue and your challenges of not seeing any packages used by said owner of the reserved prefix.

While I do think due time is ideal in many things, I think it is largely unneeded here. We aren't in the business of enforcing people to use their prefixes they requested and were granted. There should be a similar process for abandoned prefixes as you can see in the pypi issues. People email the current owner and they usually give it up, similar to package ids on any package registry. Disputes should mostly be first resolved through outside means before going to say NuGet.

People will likely try to use this maliciously and that's fine. It would make reports of abuse much more easy to make quick judgement if they could also see the history. If we had a concept of verified publishers/etc, then that would make this even more obvious. In my opinion, we should replace the current blue check mark with a label affordance instead of "Prefix Reserved" given it isn't giving the same trust perception that a blue checkmark might on other platforms.

When I first started working on this area, we added a small tooltip. It wasn't much, but it helped a little. It should also be transparent as to who owns the prefix perhaps even through clicking on that label or inside that tooltip. It should be pretty obvious to much of the community how big Microsoft / dotnet / Azure / AWS / Google are and what prefixes they might own. It would also be easier for people to see those abusing this feature today or "namesquatting".

TL;DR - Prefix reservation should be similar to a domain registrar or OSS namespace ledger. That way people don't have to waste time guessing.

[glopesdev]

@JonDouglas I actually agree with the idea of looking at this problem as a domain registrar or OSS namespace ledger, and long-term it may prove to be the only scalable solution.

The problem is that NuGet did not start out this way, and the root of the issue I am trying to identify here is how best to acknowledge that history and work through this transition with the existing community. One problem is terminology: calling it package prefix reservation obscures the fact that we are really talking about assembly namespace reservation. I don't think many long-time users realize the importance, or even the existence, of package prefix reservation, and the urgency of moving fast.

I can give examples from our own decision making. We don't even know yet why the reservation of our prefix was assigned and to whom, but we could have easily applied to register the prefix a year ago when I first learned about prefixes. Why didn't we do it? Several reasons, but most important ones rooted in the community: I looked at whether other important projects were reserving their prefixes: "FSharp", "IronPython", "OpenTK", etc, none of these have package prefix reservations; on the other hand we didn't want to damage the few other projects who had taken up our prefix while we were in the process of cleaning up and reaching out to existing users. We had recently started an effort of educating our community on the importance of not using our prefix for their plugins to start preparing this transition.

Now sadly looking back at all these concerns I feel we should have just reserved the prefix first and organized / asked questions later, but is that really the fair way to proceed? Maybe I don't know what the problem is yet, but it really feels like there is some sort of deeper issue here to be discussed, and I think it requires both educating the existing community about the importance of prefixes, and assessing and acknowledging the state of existing packages in the community during the reservation process.

It should be acknowledged that there are different contexts for the process: 1) reserving a prefix with no existing packages in it; 2) reserving a prefix in which the individual or organization reserving it has the majority of packages (and downloads); 3) reserving a prefix in which the individual has either no packages or the minority of packages. These are fundamentally different situations and they should be part of considerations for reservation, especially when the reservation application is not going to make the prefix "public".

TL;DR This essentially goes back to the ID prefix reservation criteria where it states "Would not reserving the package ID prefix cause ambiguity, confusion, or other harm to the community?" and asking the question in reverse: "Would granting the reservation to this applicant cause ambiguity, confusion, or other harm to the community?".

Finally, I do like the principle of reaching out to other owners directly and having the community self-organize. This is precisely what I did myself so many times over the years to resolve disputes. I don't think I even once bothered NuGet support because of disputes since the community was always engaging and respectful.

However, this circles back to the root cause of this issue: these new package prefix reservations can be silent, there is no "Contact Owner" / "Report Abuse" button that I can click to initiate dispute resolution other than reaching out to NuGet support.

[JonDouglas]

While I'm unsure of the specifics in your situation and perhaps someone on our team is already looped in here, I do agree in principle that it should be clear to anyone requesting a prefix to have some understanding of why a prefix may be reserved and by whom.

If you knew who(org/owner) or what packages(using it), then you could use the current process of contact/report.

Breaking down these three scenarios, here's a perspective:

1) reserving a prefix with no existing packages in it;

No different than a parked domain. You hold current rights.

2) reserving a prefix in which the individual or organization reserving it has the majority of packages (and downloads);

The individual/org can provide ample evidence in the case of a dispute.

3) reserving a prefix in which the individual has either no packages or the minority of packages.

Any person is able and welcome to reserve prefix IDs under good faith.

With all that in mind, while they may be different scenarios, the idea of reserve first, ask questions later still stands. That's why we include it in our NuGet Package Author Best Practices documentation. This is really no different than being the first to get a desired username or domain for a project.

And for what it is worth, there are many popular packages today that do not have prefix IDs reserved or probably know about the feature. Anyone could request them, but our ecosystem like many others relies on trust that people don't act in bad faith. Otherwise we have to intervene when such abuses are discovered.

[glopesdev]

@JonDouglas thank you, this is reassuring. I broadly agree with your perspective on the three scenarios. I would love to find a way to raise awareness and improve the situation for existing long-term open-source projects, but I will move any further suggestions to a new thread to avoid moving this discussion away from the open technical issue.