Add the possibility replace a nuget source (force dotnet nuget add)

[igiona]

NuGet Product(s) Involved

dotnet.exe

The Elevator Pitch

Problem statement

It is often the case that in a pipeline one has to add a nuget source with specific credentials. It is likely that such source is already registered in a nuget.config file within the repo. Simply using dotnet nuget add source "http.." doesn't work reliably, since it fails if the source is already existing.

Desired solution

Add the possibility replace a nuget source, by adding the --force argument do the dotnet nuget add source command.

dotnet nuget add source "http.." --name myNewName --force dotnet nuget add source "http.." --name myNewName --username mynewuser --password mynewtoken --force

When using the --force parameter, the dotnet command is expected to replace the source that matches the provided source-URL with the new one (of course with all parameters that might be given like username, password etc.). If not match is found, the source is just added as without the --force parameter

Additional Context and Details

Currently, a workaround to this is to remove all source with a bash command like:

dotnet nuget list source | awk '/\[/ {$1=$NF=""; print $0}' | dotnet nuget remove source {}

This command is not portable, and heavily depends on the printout of the tool which might change in future releases, making this solution pretty weak.

[kartheekp-ms]

I understand the feature request. If we know that the source is already configured in NuGet.Config file, then how about using dotnet-nuget-update-source command?

[igiona]

I understand the feature request. If we know that the source is already configured in NuGet.Config file, then how about using dotnet-nuget-update-source command?

Thanks for your reply. The point is that I can't be always sure that the source is configured, it's likely but not a guarantee. I'm sure though, that I want to have the source configured in the pipeline. As you know, dotnet-nuget-update-source fails if the source doesn't exist. Hence, the need to have a "force add" option.

[shana]

I also just ran into this issue, and would love to have a force add. Having to run a list and then figure out if the source already exists and then run either add or update depending on the result, in a safe cross-platform script way, is a lot of work, when nuget already knows all of this information and can just do it. Please add a --force option.

[Mastahh]

I have same issue on pipeline, and it is some random error sometime occurs and sometimes not, looks like that something stays in the system. I think best solution would be manually check if source already exists, and if not then add it.

[igiona]

I have same issue on pipeline, and it is some random error sometime occurs and sometimes not, looks like that something stays in the system. I think best solution would be manually check if source already exists, and if not then add it.

How can you achieve this in a robust, cross platform and maintainable way with the current tool?

[Mastahh]

I have same issue on pipeline, and it is some random error sometime occurs and sometimes not, looks like that something stays in the system. I think best solution would be manually check if source already exists, and if not then add it.

How can you achieve this in a robust, cross platform and maintainable way with the current tool?

I did this way - pwsh: > $nugetSources = dotnet nuget list source | Out-String; if(!($nugetSources -like "*nuget_source_name*")) { dotnet nuget add source "https://my_source/nuget/v3/index.json" -n nuget_source_name -u any -p "$(nugetPAT)" --store-password-in-clear-text; }

[igiona]

I have same issue on pipeline, and it is some random error sometime occurs and sometimes not, looks like that something stays in the system. I think best solution would be manually check if source already exists, and if not then add it.

How can you achieve this in a robust, cross platform and maintainable way with the current tool?

I did this way - pwsh: > $nugetSources = dotnet nuget list source | Out-String; if(!($nugetSources -like "*nuget_source_name*")) { dotnet nuget add source "https://my_source/nuget/v3/index.json" -n nuget_source_name -u any -p "$(nugetPAT)" --store-password-in-clear-text; }

Thanks for sharing!

This is similar to what I posted in the original post as workaround for linux. Your solution suffers from the same problems of mine, and that's the reason of this improvement request 👍

[Mastahh]

I have same issue on pipeline, and it is some random error sometime occurs and sometimes not, looks like that something stays in the system. I think best solution would be manually check if source already exists, and if not then add it.

How can you achieve this in a robust, cross platform and maintainable way with the current tool?

I did this way - pwsh: > $nugetSources = dotnet nuget list source | Out-String; if(!($nugetSources -like "*nuget_source_name*")) { dotnet nuget add source "https://my_source/nuget/v3/index.json" -n nuget_source_name -u any -p "$(nugetPAT)" --store-password-in-clear-text; }

Thanks for sharing!

This is similar to what I posted in the original post as workaround for linux. Your solution suffers from the same problems of mine, and that's the reason of this improvement request 👍

I removed my script from pipeline because it is not needed. I just added one action before build "Restore", for some reason Build doesn't properly do restore and not all works fine. Also I was need to return back NuGet.config into solution to have there only one source. As some investigation it is better to have nuget.config pre-configured instead of adding sources into existing one.

[rpgeddam]

@igiona Is your workaround command missing an xargs ? Or am I just proving your point that the workarounds aren't cross-platform.

[igiona]

@igiona Is your workaround command missing an xargs ? Or am I just proving your point that the workarounds aren't cross-platform.

I have the workaround successfully running on a Ubuntu-latest pipeline, so I don't think it's missing something. Maybe it could be improved via xargs though ;)

Clearly it works only on bash, hence mostly Linux. Unless you install a bash-like shell on the windows runner like git-bash or so.

[zivkan]

In case it's not well known to everybody experiencing this issue, you can use a nuget.config in your repo root, clearing out all inherited package sources, so that you know that your build pipeline will have a deterministic set of package sources configured, that does not depend on machine state: https://learn.microsoft.com/en-us/nuget/reference/nuget-config-file

<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="contoso" value="https://pkgs.dev.azure.com/contoso/vside/_packaging/internal_packages/nuget/v3/index.json" />
  </packageSources>
  <disabledPackageSources>
    <clear />
  </disabledPackageSources>
  <packageSourceMapping>
    <clear />
    <! -- add your mapping here!!!! -->
  </packageSourceMapping>
</configuration>

Modify as needed, and commit into the same directory as your .sln file (or the repo root). Then your build scripts shouldn't need to handle "maybe this package source exists, maybe it doesn't". Your CI will be more a little more resilient to unexpected machine configuration changes.

[igiona]

In case it's not well known to everybody experiencing this issue, you can use a nuget.config in your repo root, clearing out all inherited package sources, so that you know that your build pipeline will have a deterministic set of package sources configured, that does not depend on machine state: https://learn.microsoft.com/en-us/nuget/reference/nuget-config-file

<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="contoso" value="https://pkgs.dev.azure.com/contoso/vside/_packaging/internal_packages/nuget/v3/index.json" />
  </packageSources>
  <disabledPackageSources>
    <clear />
  </disabledPackageSources>
  <packageSourceMapping>
    <clear />
  </packageSourceMapping>
</configuration>

Modify as needed, and commit into the same directory as your .sln file (or the repo root). Then your build scripts shouldn't need to handle "maybe this package source exists, maybe it doesn't". Your CI will be more a little more resilient to unexpected machine configuration changes.

I already have this in use, but it does only partially work if the package registry requires authentication. Unless you want to store the authentication in your repo (which I guess nobody wants) the pipeline will still need to update the source with the necessary secret.

Thanks for sharing 👍

BTW: I would refrain from clearing the package mapping settings, to avoid opening to the "dependency confusion" attack.

[zivkan]

Unless you want to store the authentication in your repo (which I guess nobody wants) the pipeline will still need to update the source with the necessary secret.

yes, but in that case you can reliably use dotnet nuget update source, there's no possibility of the source not already existing, and therefore needing this issue's "add or update" feature request.

BTW: I would refrain from clearing the package mapping settings, to avoid opening to the "dependency confusion" attack.

I would recommend clearing package source mapping in any nuget.config that has <clear /> in the <packageSources> element, and if you want to use package source mapping, to define the mapping in the same nuget.config. The goal is to be resilient to bad machine state. If EvilHackerZivkan modifies a CI agent's user-profile nuget.config to map pattern * to a package source named nuget.org, and then creates a Directory.Build.targets file in the root directory that has a GlobalPackageReference to a malicous package, you really don't want any of those things to impact your build.

edit: it's probably safer to use a CI agent that resets fully between builds, like a docker image, or Microsoft's hosted agents. However, for self-hosted agents, I have no idea if there's the same capability to do a hard machine reset back to a base image.

[igiona]

yes, but in that case you can reliably use dotnet nuget update source, there's no possibility of the source not already existing, and therefore needing this issue's "add or update" feature request.

That's correct, it will work. This doesn't mean that this feature is not required. I prefer to tackle issues where the issue occurs in the first place. For me the pipeline should not have dependencies to the state of a config file in the repo. It's too much of a hidden dependency. I don't want to find the pipeline failing with likely some cryptic error message when a developer "messes" with the config file ;)

I would recommend clearing package source mapping in any nuget.config that has <clear /> in the <packageSources> element, and if you want to use package source mapping, to define the mapping in the same nuget.config. The goal is to be resilient to bad machine state. If EvilHackerZivkan modifies a CI agent's user-profile nuget.config to map pattern * to a package source named nuget.org, and then creates a Directory.Build.targets file in the root directory that has a GlobalPackageReference to a malicous package, you really don't want any of those things to impact your build.

edit: it's probably safer to use a CI agent that resets fully between builds, like a docker image, or Microsoft's hosted agents. However, for self-hosted agents, I have no idea if there's the same capability to do a hard machine reset back to a base image.

This is a valid point 👍 Which implies though, that after clearing the mapping, it will also be set appropriately in the repo's nuget file, which is not the case in your example above. I was more thinking of people copying your suggestion and being happy with it as is 😊

[zivkan]

I added an XML comment to my sample to point out that yes, if you want to use package source mapping, you should absolutely add it in that file. Thanks for the feedback.

[rpgeddam]

Clearly it works only on bash, hence mostly Linux.

It didn't work as is for me on mac/zsh. Adding xargs worked, but that also didn't work in my github-action running on an amzn linux 2 instance because of trailing whitespaces in the repo name. Which all goes on to reinforce the fragility of the workarounds.

I do like the repo-local nuget config file approach.

Just in case this may help someone else, the final workaround I ended up using is:

dotnet nuget list source | awk '/\[/ {$1=$NF=""; gsub(/ /, "", $0); print $0}' | xargs -I repo dotnet nuget remove source repo

[lee-11]

I've run into this as well, and would very much appreciate having --force or similar.