search

NuGet / Home

Repo for NuGet Client issues
Other
1.5k stars 253 forks source link

NuGetSdkResolver doesn't respect pivots that come through msbuild properties (example --config-file, RestoreSources) #7855

Open zarenner opened 5 years ago

zarenner commented 5 years ago

The NuGet-based SDK resolver doesn't appear to respect --configfile, and instead just uses the nuget.config next to the project.

e.g. if I have a project-level nuget.config with a single source 'sourcenexttoproject', and a separate --configfile nuget.config with a single source 'differentsource':

c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest>dotnet restore --configfile ..\blah\nuget.config
Restoring packages for C:\Users\zarenner\AppData\Local\Temp\c8ce94a0-3218-4650-ab34-1f8f61b09f3b...
c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj : warning : Unable to load the service index for source https://sourcenexttoproject/v3/index.json.
c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj : error : Unable to find package Microsoft.Build.CentralPackageVersions. No packages exist with this id in source(s): sourcenexttoproject
  Restoring packages for c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj...
C:\Program Files\dotnet\sdk\2.2.104\NuGet.targets(114,5): error : Unable to load the service index for source https://differentsource/v3/index.json. [c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj]
C:\Program Files\dotnet\sdk\2.2.104\NuGet.targets(114,5): error :   No such host is known [c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj]

The SDK is trying to restore from sourcenexttoproject, instead of from differentsource as I would expect it to. Note the sources are both invalid themselves, so the fact that there are "unable to load" errors on each is expected.

This bug completely breaks the DotNetCoreCli build task in Azure Pipelines when used against authed sources, as the task writes a temporary nuget.config with credentials and passes it via --configfile.

@jeffkl as FYI since I had discussed with him using his project SDKs for our own codebase, and this is blocking our adoption.

nkolev92 commented 5 years ago

This is becoming kind of challenge, because restore itself has various configuration pivots that likely don't work with the SDK resolver.

--config-file is one. I imagine RestoreSources & similar msbuild settings for NuGet don't work.

I'd imagine the whole list from https://github.com/NuGet/Home/wiki/%5BSpec%5D-NuGet-settings-in-MSBuild is an issue.

And that'll continue to be a moving target in the future.

//cc @rrelyea

jeffkl commented 5 years ago

Anything settings coming from MSBuild will not work with the SDK resolver because the project has not been evaluated yet. So no properties or items are loaded to even pass along. The feeds must be defined in the root nuget.config and auth tokens will have to be in that file that can be found by the SDK resolver or the credential provider needs to get them.

We could possibly plumb the custom path to the nuget.config to the SDK resolver but that wouldn't be available until 16.1 at the earliest. I would like to find a workaround that unblocks people immediately.

Doesn't the newer credential provider use environment variables to pass along auth tokens? Editing the nuget.config in a temp directory and then passing the path to dotnet.exe restore or nuget.exe seems less stable than setting env vars that the credential provider can pick up.

nkolev92 commented 5 years ago

A similar issue was raised in #8183

markwilkie commented 5 years ago

What the current state of this issue? We (.NET Core) are running into this as well and are curious what the current thinking is.

cc/ @jcagme

nkolev92 commented 5 years ago

No progress has been made on it.

Note that this scenario adds some inconvenient coupling so if you can at all avoid it, please do so.

jcagme commented 5 years ago

You are saying we should not try restoring SDK packages from a private feed?

nkolev92 commented 5 years ago

Don't specify the config file directly & allow the tooling to discover it by convention.

If you put the config file at the repo root and/or near the solution files, ti will be discovered by the tooling automatically. You can disable other cascading configs interfering with yours by adding clear tags like the ones we use in the NuGet Client repo. https://github.com/NuGet/NuGet.Client/blob/dev/NuGet.Config

jcagme commented 5 years ago

Well, at least for our case explained here we don't pass in the location of nuget.config and use the one in the root. Since we are trying to restore packages from a private AzDO feed, probably cred provider is doing that under the covers?

nkolev92 commented 5 years ago

The cred providers were enabled for the sdk resolver so that should be working the same as regular restore.

I'm not sure why it's having trouble detecting the plugin however. It should be under ~/.nuget/plugins

zarenner commented 5 years ago

I should note that the Azure Pipelines DotNetCoreCLI task does not use the credential provider today. It uses a flow where it generates a temporary config, and passes --configfile to dotnet. That means that the DotNetCoreCLI task definitely won't work today with MSBuild SDKs in authenticated feeds.

If I understand correctly, @jcagme might be seeing another issue where nuget doesn't invoke the credential properly for MSBuild SDKs. I think we need to investigate that further - I haven't had a chance to look into that yet.

jcagme commented 5 years ago

The cred providers were enabled for the sdk resolver so that should be working the same as regular restore.

I'm not sure why it's having trouble detecting the plugin however. It should be under ~/.nuget/plugins

Yes, the installation script put the plugin in ~/.nuget/plugins/netcore

I should note that the Azure Pipelines DotNetCoreCLI task does not use the credential provider today. It uses a flow where it generates a temporary config, and passes --configfile to dotnet. That means that the DotNetCoreCLI task definitely won't work today with MSBuild SDKs in authenticated feeds.

If I understand correctly, @jcagme might be seeing another issue where nuget doesn't invoke the credential properly for MSBuild SDKs. I think we need to investigate that further - I haven't had a chance to look into that yet.

Yeah, for our builds we don't use the task but use dotnet restore directly. This is what I've done:

Windows

  1. Run installation script
  2. Run build script Output: successfully restored packages in private feed

Mac

  1. Run installation script
  2. Run build script
  3. Since things didn't work, installed VS for Mac and signed in using my [at]microsoft account
  4. Run build script Output: /Users/jcagme/code/jcagme/arcade/Directory.Build.props(3,3): warning : Unable to load the service index for source https://xxx.pkgs.visualstudio.com/_packaging/xxx-test/nuget/v3/index.json. /Users/jcagme/code/jcagme/arcade/Directory.Build.props(3,3): error : Unable to find package Microsoft.DotNet.Arcade.Sdk. No packages exist with this id in source(s): dotnet-core, nuget.org /Users/jcagme/code/jcagme/arcade/Directory.Build.props(3,3): warning : The plugin credential provider could not acquire credentials. Authentication may require manual action. Consider re-running the command with --interactive fordotnet, /p:NuGetInteractive="true" for MSBuild or removing the -NonInteractive switch forNuGet

Azure DevOps build agent

  1. Run installation script
  2. Run build script
  3. Run build script Output: same as in Mac
zarenner commented 5 years ago

Is it possible your VSS_NUGET_EXTERNAL_FEED_ENDPOINTS envvar isn't getting set or is incorrect?

If it's a domain account perhaps the windows case is working because it's silently / non-interactively authenticating and thus doesn't need VSS_NUGET_EXTERNAL_FEED_ENDPOINTS? Not sure what identities are in play here for you.

jcagme commented 5 years ago

I thought that to be the case but I then included a step to echo the variable value and it is correct. It looks like {"endpointCredentials": [{"endpoint":"https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json", "password":"<BotPAT>"}]}

BotPAT is the PAT of an account I explicitly added as a contributor of the feed.

zarenner commented 5 years ago

Unable to load the service index for source https://xxx.pkgs.visualstudio.com/_packaging/xxx-test/nuget/v3/index.json

{"endpointCredentials": [{"endpoint":"https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json", "password":"<BotPAT>"}]}

pkgs.dev.azure.com/\<organization> vs \<organization>.pkgs.visualstudio.com mismatch?

jcagme commented 5 years ago

True, although https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json is what I have in nuget.config in Windows (the one which works) and also that's what I get in Azure DevOps artifacts "Connect to feed". Should I attempt https://xxx.pkgs.visualstudio.com/_packaging/xxx-test/nuget/v3/index.json?

zarenner commented 5 years ago

It must match up exactly between nuget.config and endpointCredentials. Putting both forms into endpointCredentials won't hurt.

jcagme commented 5 years ago

0>F:\workspace\_work\1\s\artifacts\toolset\restore.proj : error : Unable to load the service index for source https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json. 0>F:\workspace\_work\1\s\artifacts\toolset\restore.proj : error : Unable to load the service index for source https://xxx.pkgs.dev.azure.com/_packaging/xxx-test/nuget/v3/index.json.

It fails with both endpoints. Both endpoints live in nuget.config

nkolev92 commented 4 years ago

@jeffkl Do you think the general problem is still something we should try to address? Specifically

Anything settings coming from MSBuild will not work with the SDK resolver because the project has not been evaluated yet. So no properties or items are loaded to even pass along.

jeffkl commented 4 years ago

The only thing we could plumb through would be global properties since they are loaded before a project is evaluated. You could then set the global property via the command-line or Directory.Build.rsp. We will never be able to use project properties in the SDK resolver since there is no way to "peek" ahead.

The issue for this proposed feature is here: https://github.com/microsoft/msbuild/issues/2095

ghost commented 3 years ago

Note, posted stand alone repro in item https://github.com/NuGet/Home/issues/10886

@jeffkl , any news on this?

jeffkl commented 3 years ago

I do not think we'll be adding support for this any time soon. The workaround is to just check in your NuGet.config and use the same sources for all builds. The ability to dynamically use different sources at build time is just not a very common scenario. If you want to dynamically change sources at build time, you'll need to write a script that changes the NuGet.config before restore and build.

ghost commented 3 years ago

I'm aware of the workaround, but it wastes developer time to have a documented feature that doesn't work.

Would highly recommend updating the docs for --config-file to indicate that it doesnt' work for Sdk imports.

ghost commented 3 years ago

Also, consider piping the appropriate config parameters as environment variables. E.g. add NUGETSDKRESOLVER_CONFIG_FILE and update dotnet.exe --config-file to configure it appropriately.

Pvlerick commented 2 years ago

Also, consider piping the appropriate config parameters as environment variables. E.g. add NUGETSDKRESOLVER_CONFIG_FILE and update dotnet.exe --config-file to configure it appropriately.

Having an env variable would indeed be helpful...

m0sa commented 2 years ago

Running into the same issue, the RestoreConfigFile MsBuild property / env variable is not being picked up.

amaltinsky commented 2 years ago

I'm aware of the workaround, but it wastes developer time to have a documented feature that doesn't work.

Would highly recommend updating the docs for --config-file to indicate that it doesnt' work for Sdk imports.

I just wasted several hours tracking down this issue in a CI pipeline. I'd go one step further than the documentation and suggest changing the error message shown by NuGet in this case from "error MSB4236: The SDK .. specified could not be found" to something more explicit about NuGet not being able to restore SDKs when --config-file option is provided.