NuGet Restore/Update does not work when unrequired package sources are unavailable

[deadlydog]

Details about Problem

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe): VS UI

NuGet version (x.x.x.xxx): NuGet Package Manager 4.5.0

VS version (if appropriate): VS 2017 (v15.5.2)

OS version (i.e. win10 v1607 (14393.321)): Win 10 (v10.0.16299 Build 16299)

Worked before? If so, with which NuGet version: No

Detailed repro steps so we can see the same problem

1. Add an additional package source that can only be accessed from the local network (or when connected via VPN).

2. Open up a solution in Visual Studio that only uses packages from nuget.org (not from the LAN package source).

3. Attempt to restore packages, or open the NuGet Package Manager UI with "nuget.org" as the "Package source" and attempt to update the packages, when the LAN package source is not available.

4. You will receive an error like:

Exception 'System.AggregateException' thrown when trying to add source 'https://packages.mydomain.com/nuget/nuget'. Please verify all your online package sources are available.

So even though the packages that you are trying to update/restore are not on the alternate package source, NuGet still wants that package source to be available before continuing. This means you either need to get on the LAN that has the package source (or connect to the VPN), or you need to go disable/remove that package source from your list of package sources.

This happens both when using the NuGet Package Manager UI and the Package Manager Console.

Other suggested things

I think a better approach would be to try and restore/update all of the packages, and if they can't be found report an error and have that error mention that it could not connect to the alternative package source, and that may be the source of the problem.

This is a problem for many people I work with, as we have an internal corporate nuget package source. Anytime we work remotely or from home, even on projects that don't use packages from our corporate source, we need to connect to the VPN in order to restore/update packages.

Verbose Logs

PM> Update-Package -Verbose
  GET https://api.nuget.org/v3/registration3-gz-semver2/microsoft.azure.mobile.analytics/index.json
  OK https://api.nuget.org/v3/registration3-gz-semver2/microsoft.azure.mobile.analytics/index.json 436ms
Update-Package : Unable to load the service index for source https://packages.mydomain.com/nuget/nuget.
  An error occurred while sending the request.
  The remote name could not be resolved: 'packages.mydomain.com'
At line:1 char:1
+ Update-Package -Verbose
+ ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Update-Package], Exception
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.UpdatePackageCommand

Time Elapsed: 00:00:00.6385733

[StingyJack]

I've logged this a few times and it gets closed with reason "working as designed".

https://developercommunity.visualstudio.com/content/problem/69961/manage-nuget-packages-for-solution-is-not-paying-a.html

https://github.com/NuGet/docs.microsoft.com-nuget/issues/407

https://developercommunity.visualstudio.com/content/problem/118566/vs-is-failing-to-install-a-nuget-package-from-nuge.html

[zhili1208]

this is by design, nuget restore hard fails when any enabled source is not available, we don't want to restore inconsistently based on which source is available. Please disable unused source before restore.

[StingyJack]

@zhili1208 - please explain how a restore can in any way be inconsistent in the scenario described above.

[StingyJack]

@zhili1208 - if I try to push a package to a nuget.server and that package has dependencies, but the server does not have those dependencies yet, will the push succeed?

[rlundy]

@zhili1208 This continues to baffle me. It baffles me, in fact, every day when VS fails to install a package or fails to update a package or fails to create a flippin' controller in an ASP.NET Core MVC project because a NuGet package source that I'm not using and don't care about is unavailable.

Instead of continually closing this as "by design," what about a little introspection? What about examining a few alternatives? This keeps getting opened by people because it's a constant irritation. Essentially NuGet is deciding that it's smarter than we developers are, and that it must protect us from ourselves. Please fix this so that NuGet will ask whether I actually care that a package source is unavailable before it fails.

[deadlydog]

I have to agree with @rlundy. I understand that you are moving towards a new PackageReference model and looking to improve the design in that "vNext" model, but the reality is that many people will continue to use the "old" packages.config model for many years yet. Adding a small UI enhancement to prompt users if they want to continue when a package source is unavailable, or simply attempting to get/upgrade packages and only display an error if there is a failure would go a long way toward a better user experience for developers that will be continuing to use VS 0217 and .Net Full (as opposed to .Net Core) for the next 5 years.

~@StingyJack~ @zhili1208 please reconsider re-opening this issue. Thanks.

[StingyJack]

@deadlydog - i didn't close it, @zhili1208 did. I've complained about the same thing a few times and get stonewalled because "muh transitivity" or "muh package source diagnostics".

The post I linked above is an actual example of how adding a package with dependencies that are not already present on the nuget server is blocked. This neuters the argument of one sources not being available could cause dependency issues, as they all must be on the same server anyway

Basic stuff doesn't work right with nuget and instead of fixing it, msft is interested in what is generally the most expensive option in software development - to trash something and start over, instead of addressing the gaps.

[StingyJack]

@zhili1208 or @rrelyea - if this is closed "by design" then you probably have a bug in your design.

[rrelyea]

On phone currently, so not easy to find backlog item...

1) believe in the need to require all feeds 2) can help users disable sources when offline — has a design, but have not got to that. 3) we should definitely make sure we have a clear error experience, which seems to be questioned by nick craver in this Tweet: https://twitter.com/nick_craver/status/1021714890640289792?s=21

@zhili1208 - for #2, please find backlog item and add link to this comment. For #3, is our error experience clear enough? Please chat offline with me too

[NickCraver]

FWIW, we hit this constantly as we have a feed behind the VPN. We've also learned various ways to produce a minimal repro quickly. I'd be more than happy to chat with anyone on the team about fixing it.

[StingyJack]

On phone currently, so not easy to find backlog item...

believe in the need to require all feeds

@rrelyea - you cannot mean that Package A from Feed A cannot be installed when Feed B is unavailable, and when Package A has no immediate or ancestral dependency in Feed B.

That dependency scenario cant happen anyway currently, as you cannot push Package A to a Feed A unless all its dependencies are already present in Feed A. EDIT: Feed A and B are private feeds using Nuget.Server

[jainaashish]

As I mentioned on https://github.com/NuGet/Home/issues/7015 this issue simply doesn't mean that we start to ignore failing sources while restoring (please read the full comment on #7015).

About the dependency scenario, it's quite easy possible that packageA is coming from private feedA but depends on packageB which is coming from public feedB. We've already seen several such cases for popular packages which often customers don't push to their private feeds rather directly bring them from public feeds.

So until we have some way to lock down package versions and each restore will resolve to the same version independent of feeds then this feature should be enabled to ignore failed sources in UI.

[StingyJack]

you cannot mean that Package A from Feed A cannot be installed when Feed B is unavailable, and when Package A has no immediate or ancestral dependency in Feed B. (me)

About the dependency scenario, it's quite easy possible that packageA is coming from private feedA but depends on packageB which is coming from public feedB. (you)

We are stating different scenarios.

While the dependency graph is built and nuget is querying sources to fill that graph, if it finds what it needs in one source, why should it demand the other sources be available? It seems inefficient.

If I cant update a public package from nuget.org, because the VPN isnt connected at the moment and cant reach a private feed (a feed that is unavailable, not failing) it is going to be viewed as an artificial and needless impediment.

[rlundy]

I understand that there's a real reason for this restriction to exist.

But there are also real reasons for this restriction not to always exist.

NuGet can already pop up dialogs; it does so for license acceptance.

I think all we're looking for is for NuGet to pop up a dialog saying "Hey, I can't reach feed XYZ; should I proceed anyway?"

Just for review, the use case is this:

In addition to the NuGet public feed, I use a private feed from my company that I connect to by VPN. If I'm not connected to the VPN, that private feed is unavailable. In that case, NuGet simply refuses to work until I either connect to the VPN, or disable that package source (and then I have to remember to re-enable it later...and every time I do anything with NuGet on this project I have to disable it again).

If I'm working on a project that has nothing to do with my company's private NuGet feed, NuGet shouldn't refuse to work without me connecting to that unavailable feed. Or at least I should be able to tell NuGet "I don't care that I can't reach that feed" without having to continuously disable/enable feeds.

[goldsam]

How about adding an argument or switch to override the default behavior? This seriously kills productivity. I think there is a pretty clear community sentiment that the current behavior is unhelpful at best. To be honest, its downright scary that a single point of failure was intentionally introduced by design into mission critical tooling.

[rlundy]

Proposal for how to fix this: A configuration file that I can put in my project root with a list of NuGet feeds I want to ignore for this project.

For a new project, I'd create this file and put a list of NuGet feed URLs in there that I never want to use with this project. (E.g. this is a hobby project, so I want to ignore my company's NuGet feed which is annoyingly behind a VPN)

It would mean I'd have to create this file once for each new project, but there's a fair bit of setup anyway for a new project, so I don't think this would be an unreasonable burden.

Maybe it could be a .nugetignore file. 😁

[StingyJack]

@rlundy - does this work to meet that need? https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file#disabledpackagesources

[rlundy]

Holy moley!

I had no idea such a thing existed.

How did nobody mention it till now in a year and a half of this issue existing!?

Thanks, @StingyJack! It looks like precisely what I'm looking for. I'll have to give it a spin later.

[deadlydog]

Adding the disabledPackageSources is a potential work around, but still a very painful one. I shouldn't have to edit all of my personal (and open source) projects to tell them to ignore my work's internal NuGet package source just so I can restore NuGet packages when not connected to my work's VPN.

[dquist]

👋 +1 from me as this issue has been constantly impacting our team since we set up a private nuget feed to host internal packages. There's literally no reason nuget should require a private feed to exist if I'm explicitly telling it to install a public package from nuget.org. What a nuisance!

[agaace]

My biggest issue with this behavior is that it causes obscure failures if you use Visual Studio to build your projects. I opened this issue for Visual Studio: https://developercommunity.visualstudio.com/content/problem/878141/nugget-restore-randomly-failing-with-obscure-error.html and they closed it as a dupe of this issue.. So we're in a downward spiral of confusing/obscure behavior here.

[sharprs83]

This needs fixed. To have to go to Tools > Options to disable a source temporarily while not connected to my work VPN just to use a public package that can't depend on a private package is adversarial UX design.

[rlundy]

For those who missed it above...

WORKAROUND

From a command prompt:

> nuget sources

It shows you something like:

Registered Sources:

  1.  nuget.org [Enabled]
      https://api.nuget.org/v3/index.json
  2.  My Company's NuGet Server [Enabled]
      https://example.com/whatever/nuget
  3.  Microsoft Visual Studio Offline Packages [Enabled]
      C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

Now create a file called nuget.config. Put this in it:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <disabledPackageSources>
    <add key="My Company's NuGet Server" value="true" />
  </disabledPackageSources>
</configuration>

Put that file in the root of your non-company code folder.

Done!

To confirm, run nuget sources in that same folder and you should now see [Disabled] next to whichever NuGet sources you put in the file.

There are other ways to do this, such as globally disabling your company's NuGet source on your PC and then only enabling it for the folder where you keep your company's code. These methods are left as an exercise for the reader. 😉

Note: You do not need a nuget.config for every project! You only need it in the root folder and it cascades. If your situation is more complicated, you can put nuget.config files at any level and change the enable/disable stuff as necessary.

[innodonni]

A simpler solution I found by accident, for those starting a new side project and planning to create a private NuGet repo for it later, is to run dotnet new nugetconfig, which will clear the inherited sources and add just the nuget.org one.

[BlueManiac]

Why is this still a thing? It's very weird that I must disable our company nuget just to be able to run ex. dotnet tool install -g dotnet-aspnet-codegenerator There clearly needs to exist a better solution than the current one.

[ghost]

Issue is missing Type label, remember to add a Type label

[kev160967]

I'm not sure if this is the same issue or not. I've got an ADO build pipeline that occasionally gives me this error, then works perfectly if I just start the pipeline again. It issues the following command and gets numerous variants on the error you can see, before giving up:

C:\agent\_work\_tool\dotnet\dotnet.exe publish C:\agent\_work\8\s\Saturn.Main\Saturn.Main.vbproj -c Release --runtime win-x64 --self-contained /p:PublishSingleFile=false --output C:\agent\_work\8\a\Saturn.Main
MSBuild version 17.3.2+561848881 for .NET

  Determining projects to restore...

  Retrying 'FindPackagesByIdAsyncCore' for source '[https://packages.infragistics.com/nuget/licensed/FindPackagesById()?id='Microsoft.WindowsDesktop.App.Runtime.win-x64'&semVerLevel=2.0.0'.](https://packages.infragistics.com/nuget/licensed/FindPackagesById()?id=%27Microsoft.WindowsDesktop.App.Runtime.win-x64%27&semVerLevel=2.0.0%27.)

  Response status code does not indicate success: 400 (Bad Request).

The project has two feeds defined in nuget.config, nuget.org and the infragistics feed, but the publish command either checks both or checks the infragistics one first and fails. I'm wondering if this could be down to the infragistics source being momentarily unavailable

[zivkan]

This comment is by no means claiming this issue is "solved" or shouldn't have product changes made, but there are some features, some new, some not, which can reduce the impact

disabled package sources

As mentioned by others earlier in the thread, it's possible to have package sources disabled in nuget.config, and nuget therefore won't attempt to contact the server during restore or package install. It can be done in VS's package source options, or on the command line with dotnet nuget disable source <name> (check with dotnet nuget list source from your solution directory). If you don't have a repo nuget.config, I STRONGLY recommend that you do, if for no other reason that build reliability & security (if <packageSources> starts with <clear /> then it won't inherit package sources from other nuget.config files, and NuGet won't contact package sources from shared machine state)

ignore failed sources

Earlier in the issue, there is mention of "ignore failed sources", but the text reads as if it wasn't implemented at that time. Well, now dotnet restore --ignore-failed-sources exists. If your project uses floating versions, you might get weird results, but it's an option.

Disable NuGetAudit

Starting with VS 17.8 and .NET 8.0, there's a feature called NuGetAudit which checks packages for known vulnerabilities, by trying to get a list of packages with known vulnerabilities from all http(s) package sources. Use dotnet restore -p:NuGetAudit=false or set <NuGetAudit>false<NuGetAudit> in an appropriate MSBuild file as a property.

Package source mapping

NuGet added Package Source Mapping in the last year or two. The simplest case is if all your company internal packages start with your company name, then I can configure my contoso feed with prefix contoso.*, and then NuGet will only attempt to contact the feed when a package with that id prefix is needed. Therefore, I should be able to install and upgrade packages from nuget.org while disconnected from the VPN, because all the contoso.* packages my project needs are already in my global packages folder.

[levicki]

@zivkan

NuGet added Package Source Mapping in the last year or two. The simplest case is if all your company internal packages start with your company name, then I can configure my contoso feed with prefix contoso., and then NuGet will only attempt to contact the feed when a package with that id prefix is needed. Therefore, I should be able to install and upgrade packages from nuget.org while disconnected from the VPN, because all the contoso. packages my project needs are already in my global packages folder.

As I said in the other issue which you closed as a duplicate of this one that doesn't work as explained (and expected). Hsa that been fixed in the meantime?

[StingyJack]

Disable NuGetAudit

@zivkan - is this an option that can be set in visual studio or would I have to add the property to every proj file?

[zivkan]

As I said in the other issue which you closed as a duplicate of this one that doesn't work as explained (and expected). Hsa that been fixed in the meantime?

In your original post, before I closed as a duplicate, there was no mention of package source mapping. Your reply in that issue also didn't indicate any actual experience with bugs in Package Source Mapping, only a comment which honestly I read as somewhat passive aggressive. You asked if we tested it, and we very clearly do test package source mapping.

If you're finding that Package Source Mapping is downloading packages from a feed that the configuration shows it should not, then there's a gap in our unit tests, but please create a new issue because that's a separate issue from the title of this issue. Preferably with a sample solution that we can use to run locally. If you look at the NuGet.Client repo nuget.config, you can find multiple public nuget feeds, the VSIDE feed in particular has some packages that do not exist on nuget.org, so it should be possible to provide a repro that uses only public feeds. You can use the NUGET_PACKAGES environment variable to use a different global packages folder (GPF), so that you can more easily clear it between every test restore, since package source mapping doesn't override NuGet's behaviour of not downloading packages that are already in the GPF.

Disable NuGetAudit

is this an option that can be set in visual studio or would I have to add the property to every proj file?

The only configuration is from MSBuild properties, so there's no Visual Studio or nuget.config setting. But as per MSBuild rules, command line properties (global properties) override all other values, so for CLI scenarios -p:NuGetAudit=false will override any other value, including any explicit <NuGetAudit>true set in any MSBuild file. Within a repository, or directory structure, Directory.Build.props can be used to set the value in one place and have it apply to all projects in subdirectories. But it will need to be configured for each repo separately, although MSBuild does also provide extensibility to customize all builds on a computer. I wish customers didn't need to learn about MSBuild, but unfortunately it is what it is.

[levicki]

In your original post, before I closed as a duplicate, there was no mention of package source mapping.

If we are talking about the same post (the issue I created which you closed as a duplicate of #6373), then what you just said is totally incorrect. Here is a screenshot of the bug report with relevant details about package source mapping circled in blue pen:

... only a comment which honestly I read as somewhat passive aggressive.

If we are being honest then let me tell you that I am finding Microsoft developers' responses to bug reports from paying customers to be ranging from passive-aggressive to outright abusive. If you took the time to read the bug report and if you tried to reproduce instead of dismissing it then there wouldn't have been necessary to comment further.

If you're finding that Package Source Mapping is downloading packages from a feed that the configuration shows it should not, then there's a gap in our unit tests, but please create a new issue because that's a separate issue from the title of this issue.

I wrote my findings in an issue which was closed, I have no desire to waste my time repeating myself. The only reason I posted the comment here is because from my testing your claim:

and then NuGet will only attempt to contact the feed when a package with that id prefix is needed

Is not true.

Take from that what you want, I am done with providing feedback for free only for it to be ignored.

[zivkan]

If we are talking about the https://github.com/NuGet/Home/issues/12950 (the issue I created which you closed as a duplicate of https://github.com/NuGet/Home/issues/6373), then what you just said is totally incorrect.

Sorry, you're toally right. I remember that I used ctrl-f in the browser to try to find source mapping, so I must have been looking at the wrong issue at the time. Or maybe I searched the wrong keyword. I don't know. Some kind of dumb human mistake in any case. I reopened that issue because I agree that with source mapping enabled, I believe it shouldn't cause failures in restore or adding a new package that comes only from available sources.