NuGet cannot restore from HTTPS sources that have SSL certificate problems

joelverhagen commented 7 years ago

Idea

We should add a way for users to bypass certain HTTPS certificate errors. If the user has a source that does not have a valid certificate chain, they should still have some way of getting NuGet to interact with this source.

"Inspirations"

npm and yarn have this:

npm config set strict-ssl false
yarn config set strict-ssl false

Maven has some extra knobs and switches:

-Dmaven.wagon.http.ssl.allowall=true
-Dmaven.wagon.http.ssl.insecure=true
-Dmaven.wagon.http.ssl.ignore.validity.dates=true

PHP's Composer

{
    "config": {
        "disable-tls": true,
        "secure-http": false
    }
}

curl has it:

--no-check-certificate

Specific Scenario

nuget.org server side runs functional tests against the staging slot of DEV, INT, and PROD that use nuget.exe. The DNS for this staging slot is some random blah.cloudapp.net. The certificate used is associated with the production DNS (e.g. *.nuget.org). This means nuget.exe fails SSL checks when running against this staging slot.

Proposal

Add a NuGet.config trustedCertificates section with standard <add> and <remove> tags. The value should be the SHA1 fingerprint (a.k.a. thumbprint) of the certificate. This allows developers to specifically opt-in into the tenuous state of SSL validation failures. The risk is mitigated since the developer must opt into it and even then only on a case-by-case basic.

Any certificate added to the trustedCertificates set is excluded from SSL validations -- I am thinking ALL validations exposed to use via the .NET API (untrusted root, name mistmatch, etc).

To improve discovery, we can add a log which writes out the fingerprint of the certificate that failed SSL validation at the error level.

We should also consider adding an environment variable to set this value. I believe npm has this and this sibling environment variable would match other NuGet.config keys, such as the location of the user packages directory.

Implementation

HttpClientHandler in System.Net.Http 4.3.0 (maybe earlier) has a way to set the certification validation check. This allows an app to configure the certificate validation on a per-request basis, instead of a per-AppDomain basis previously provided by ServicePointManager static properties.

With this approach, we should be able to implement the settings for all of our entry points: nuget.exe, .NET CLI, and (most tricky) Visual Studio.

The only possible problem is TFM compatibility with the System.Net.Http package. This requires investigation.

Issues

scottbommarito commented 7 years ago

Is there any danger for us to add an "allow all SSL certs" configuration much like the other package managers? I understand that it is a risky operation that somebody shouldn't do, but given that so many others do it, I could imagine somebody being frustrated that we don't allow the functionality.

joelverhagen commented 7 years ago

Good question. I would rather be reactive to that change based on community requirements. This feature request is already driving by an "external" requirement (server team). I think a broader option is unnecessary and even undesired for us. If a customer wants that broader SSL validation opt-out, they should speak up 😄.

gavinbeatty commented 7 years ago

For the problem of corporate firewalls which MITM traffic with their own CA, a solution other package managers/apps use is to add a config option to use a specific CA bundle, or respect the environment variables SSL_CERT_DIR or SSL_CERT_FILE. Examples:

~/.curlc: cacert = /path/to/ca
~/.wgetrc: ca_certificate=/path/to/ca
~/.pip/pip.conf:
```
[global]
cert = /path/to/ca
```
rustup: respects SSL_CERT_FILE
git: http.sslCAInfo

I think this suggestion might be orthogonal to yours, @joelverhagen. I don't know how common it is for certs to be broken -- in my case, I just want NuGet to recognize the unknown cert that is valid.

joelverhagen commented 7 years ago

@gavinbeatty wouldn't the Windows-y solution here be to install the corporate CA's certificate as a trusted root on that machine?

KyleWiering commented 7 years ago

@joelverhagen Correct, that is the way to remedy the issue of a self signed cert not being recognized. as @gavinbeatty suggests, most other utilities that connect through ssl provide an optional path to a cert bundle - as they do not know how to reference the Windows Certificate store.

Newer composer uses PHP.ini openssl.cafile=path/to/cafile

The suggestion really is an approach to getting stuff working quickly in a local environment - and bringing nuget more in-line with other package managers. The downside to such ability, as @scottbommarito suggests, is that it can be used to turn off TLS inappropriately.

Not being able to temporarily ignore a cert signature is a frustration. Admittedly, I come from a more open source / composer / npm background - and internal proxy package hosts with self signed certs used congruously with the approved external package hosts is commonplace.

pcewing commented 7 years ago

Just a note, I came across this issue after trying to restore packages with NuGet on Ubuntu via the Rider IDE. It failed despite having trusted certificates installed via the ca-certificates package. Having the option to either bypass SSL or specify trusted certs via environment variables would be great!

joelverhagen commented 7 years ago

@pcewing, what would the ideal configuration be for you? I see three options so far:

Allow (trust) all SSL certificates. This seems like @scottbommarito's suggestion.
Allow a "cafile" option where you can point at a file containing one or more trusted certs. Presumably these are trusted roots not just specific certificates. This seems to be @gavinbeatty and @KyleWiering's suggestion.
1. Maybe supporting well known environment variables SSL_CERT_DIR and SSL_CERT_FILE.
Allow trusting of specific certificate fingerprints (thumbprints) via NuGet config. This does not imply trusting a root just trusting specific SSL certificates. This is my suggestion.

I think one or more of these could be supported at the same time. I want to get a sense of what covers different people's scenarios. Note although I am on the NuGet team, this feature is by no means a commitment -- we're in the brainstorming phase here 😄!

pcewing commented 7 years ago

@joelverhagen I think a combination of 1 with something like 2 or 3 would be great. Another package manager I frequently use for Erlang/Elixir is Hex, and they added support for a HEX_UNSAFE_HTTPS environment variable that can be set when SSL should be bypassed altogether.

Personally I think option 2 sounds better as setting environment variables is already a standard step when configuring build/CI machines and it's easy for an IT department to do globally on developer workstations. Option 3 seems more useful in the case where a repository specifically depends on trusted certs, so every developer who clones that repo automatically gets the Nuget.config pointing to them.

Either way, just having a mechanism would be nice. With .NET core being adopted and development becoming more common on other operating systems, it will probably become a more frequent request. And don't worry, I totally understand! :)

gavinbeatty commented 7 years ago

There are many solutions that might work:

if the software looks in the Windows cert store, install the cert if you have permissions.
if that doesn't work out (unfortunately it is quite common for non-Microsoft open source software to not look in the Windows cert store at all):
- ignore all cert checks (option 1).
- ignore specific cert checks (option 3).
- specify an alternative cert (option 2).

I'm assuming NuGet does the right thing and looks in the Windows cert store, but I still slightly prefer option 2, the alternative cert file/dir (my suggestion).

From memory, I think there are "personal" and "system" cert stores in Windows and it's not obvious which to choose.
No permissions required.
Actually checks certs.
Everybody else is doing it. I think it's likely that people other than me have already been searching Stack Overflow for "how to specify ssl cert for nuget", etc., simply because that's the goto solution in other domains. It would be nice if the answer was a straightforward "we have that in nuget too".

foresightyoue commented 7 years ago

@joelverhagen I have the nuget problem by CA certificates as below,could it be fixed by engored the CA certificates?if so,how to set the nuget config with egnored the CA certificates?Thanks.

user@a1adaf3:/projects$ dotnet restore log : Restoring packages for /projects/CSharpDemo/project.json... error: Unable to load the service index for source https://api.nuget.org/v3/index.json. error: An error occurred while sending the request. error: Peer certificate cannot be authenticated with given CA certificates

joelverhagen commented 7 years ago

@hwx405562, today NuGet only supports using the built-in set of CA certificates. This behavior is defined by the implementation of .NET Core's HTTP stack. On Windows, this is the system's trusted CAs from the Windows certificate store. I am not sure about the specifics of macOS and Linux.

This issue is mainly focusing on designing a new feature. Could you file a new issue with your question, including OS version, .NET CLI version, repro steps, etc? We can focus on your particular issue on more depth with a seperate GitHub issue. https://github.com/NuGet/Home/issues/new

gavinbeatty commented 7 years ago

It sounds like @pcewing and I would both be very pleased with environment variables, but @joelverhagen, what are your thoughts?

anaselhajjaji commented 7 years ago

In my side, I forked the nuget and modified the source code to accept all the certificates (it's for internal use in company). https://github.com/anaselhajjaji/nuget/tree/db4c607f1b326074da583e9a13bae47932d963f4/nuget-binary

zhili1208 commented 7 years ago

close as dupe of #5773

emmellee commented 6 years ago

@zhili1208, @joelverhagen, @rrelyea, could I persuade you and your nuget team members to take a renewed interest in support for ssl certificates? My organization wishes to create nuget packages for release to our internal feed provided by our TFS Server. The DoD relies heavily on the use of smartcards and CA certs and, like any DoD webserver, our TFS Server requires mutual ssl authentication. Unfortunately, we are unable to get past this and duplicate issue #5773. Any help your team can provide towards resolving this issue would be greatly appreciated. The TFS/VSTS teams have offered their resources towards resolving their piece of the puzzle (reference issue #5773). Thank you.

tmds commented 6 years ago

Maybe supporting well known environment variables SSL_CERT_DIR and SSL_CERT_FILE.

These variables are supported by .NET Core 2.1.

alexei-matveev commented 6 years ago

Some may be affected by Dotnet SDK using too specific SONAMEs for libssl. I just had to troubleshoot somewhat similar problem. Dotnet picks some old libss.so.1.0.0 from a random directory whereas most executables refer to libssl.so.10 or libssl3.so on CentOS 7:

$ strace -e open dotnet restore 2>&1 | grep libssl open("/lib64/tls/libssl.so.1.0.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/lib64/libssl.so.1.0.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libssl.so.1.0.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib64/libssl.so.1.0.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib/nsr/lib64/libssl.so.1.0.0", O_RDONLY|O_CLOEXEC) = 79

Compare that to the output of

ldd /usr/bin/* | grep libssl

that is dominated by

    libssl3.so => /lib64/libssl3.so (0x00007f3584227000)
    libssl.so.10 => /lib64/libssl.so.10 (0x00007f78c8b66000)

This machine happens to have some legacy software installed that brings its own libssl.

alexei-matveev commented 6 years ago

With the newest version of SDK mine case is closed:

$ dotnet --info .NET Core SDK (reflecting any global.json): Version: 2.1.302 Commit: 9048955601

awnugent commented 5 years ago

Just in case this is helpful to others, I found that, in an Azure DevOps on-prem agent context, that simply disabling the Install NuGet task and relying on the nuget.exe installed with the ADO agent seemed to work. I consider this a workaround and NOT a fix. MITM certs in sensitive, security conscious environments is a real concern and a real fix would be great. We actually have the MITM device certs in our chain and on the system in question, but apparently nuget doesn't respect their authority ;) PLZ make better.

tavisca-cchowdhary commented 4 years ago

If you are having this error in a docker container then run

docker system prune --volumes.

Then stop the docker app in your machine. Restart your machine and running everything from start.

cs101 commented 4 years ago

being able to specify SSL exceptions in an explicit way is valuable. especially in enterprises that use MITM certs like Zscaler.... I've being fighting this in my most recent two jobs. if Nuget doesn't provide a way to do it it's going to lead a a lot of risky work around anyway.

BogusDude99 commented 4 years ago

Agreed, please give an option to bypass cert checks for the corporate MITM scenarios

fsdrw08 commented 4 years ago

my current company also had a MITM security gate (bluecoat), same issue, when run dotnet restore, error pops up, The remote certificate is invalid according to the validation procedure.

Please provide a dotnet cli option to disable or bypass ssl validation... thanks

JakeDyme commented 3 years ago

I'm struggling with this for several days now and I'm about to give up on using this stack (Docker + Linux + dotnet + AzureArtifacts) thanks to Nuget alone. We have Netskope in our org which uses self-signed certs. Both nuget and dotnet refuse to accept the certs. Npm and Yarn have both succeeded at using the Azure repo thanks to the ssl overrides, but Nuget is just dead in the water. So many threads on Stack overflow for this problem. Is there a working solution yet?

ronnyek commented 3 years ago

I've just got one question... I've got devs complaining about this... basically we have a corporate level self signed cert authroity and host our own azure devops server with its artifact feeds. User is complaining of errors with ssl trust only from the nuget CLI.

Should nuget.exe be trying to use certs that are trusted in windows cert store, or should they be somewhat disconnected, and there is some other way I should say these are also trusted certs or something?

zivkan commented 2 years ago

GitHub notifications tells me this issue was updated in the last hour, but I only see a comment from 6 months ago 😕

Anyway, since that question was never answered: NuGet uses standard .NET APIs to validate certificates. Therefore any certificate that the "operating system" trusts will be fine with NuGet (Trusted root CA store)

zivkan commented 2 years ago

Actually, the "NuGet uses standard .NET APIs to validate certificates" refers to signed package validation.

For HTTPS, NuGet uses HttpClient, without any special configuration, so if a console app using await (new HttpClient()).GetAsync("https://your-host/") works, then so should NuGet.

tparikka commented 2 years ago

My team is impacted by this issue as well - we use Zscaler on site and need a way to provide the root MITM cert as trusted.

chrisdot commented 2 years ago

Have the same kind of scenario: hosting azure devops nuget artifacts on an on prem server with a kind of self signed cert (don't know exactly what the IT team did). I have no way to trust the cert in case I'm building a asp.net core app in a docker container, because it seems it does not use the host machine's installed trusted certs. My colleague focusing on node/npm applications does not have the same problem as he has an option with npm to trust the certificate anyway.

kartheekp-ms commented 2 years ago

It looks like HttpClientHandler.DangerousAcceptAnyServerCertificateValidator Property will be helpful while addressing this issue.

jeffkl commented 2 years ago

karpikpl commented 1 year ago

Is this still without solution? I can curl and wget but nuget fails every time.

building a docker image with custom nuget feed in a MITM corporate scenario...

How should I install the MITM cert so that .Net and nuget are OK with it?

marco-carvalho commented 1 year ago

right now I can't develop with dotnet 6/7 using docker inside my organization because of this really wish there was a way to disable it like npm/yarn

edit: in my company the problem was kaspersky, we put "*.nuget.org" in the whitelist and it's working

Octarines commented 1 year ago

I want to throw my hat on the pile for this. Working in a corporate environment using zScaler (which uses MITM certs) and cannot locally run a docker build because the solution utilises a custom nuget feed.

marco-carvalho commented 1 year ago

in my company the problem was kaspersky, we put "*.nuget.org" in the whitelist and it's working

karpikpl commented 1 year ago

I want to throw my hat on the pile for this. Working in a corporate environment using zScaler (which uses MITM certs) and cannot locally run a docker build because the solution utilises a custom nuget feed.

You can add this to Docker layer that executes dotnet restore

RUN openssl s_client -showcerts -connect nuget.org:443 </dev/null | sed -n -e '/-.BEGIN/,/-.END/ p' > /usr/local/share/ca-certificates/ZscalerRootCertificate-2048-SHA256.crt && \
    update-ca-certificates

Octarines commented 1 year ago

I want to throw my hat on the pile for this. Working in a corporate environment using zScaler (which uses MITM certs) and cannot locally run a docker build because the solution utilises a custom nuget feed.

You can add this to Docker layer that executes dotnet restore
RUN openssl s_client -showcerts -connect nuget.org:443 </dev/null | sed -n -e '/-.BEGIN/,/-.END/ p' > /usr/local/share/ca-certificates/ZscalerRootCertificate-2048-SHA256.crt && \
  update-ca-certificates

I've tried this (as well as a number of similar approaches) to no avail.

hvojdani commented 1 year ago

I think nuget cli needs an option to set certificate for each source, so people can deploy their nuget servers by self signed cert. on push or get if server cert was different from what user specified cert, throw an error. its logical and secure way that most of developers are ok with that.

noeltupas commented 1 year ago

I want to throw my hat on the pile for this. Working in a corporate environment using zScaler (which uses MITM certs) and cannot locally run a docker build because the solution utilises a custom nuget feed.

You can add this to Docker layer that executes dotnet restore
RUN openssl s_client -showcerts -connect nuget.org:443 </dev/null | sed -n -e '/-.BEGIN/,/-.END/ p' > /usr/local/share/ca-certificates/ZscalerRootCertificate-2048-SHA256.crt && \
    update-ca-certificates
I've tried this (as well as a number of similar approaches) to no avail.

I tried our cert for bypassing Zscaler. No luck either.

Octarines commented 1 year ago

I want to throw my hat on the pile for this. Working in a corporate environment using zScaler (which uses MITM certs) and cannot locally run a docker build because the solution utilises a custom nuget feed.

I finally found my solution and it ended up being a combination of a lot of things. The last hurdle was because my corporate certificate was in der format rather than pem (which seems to be needed for the debian base which is used by the dotnet/sdk image). The use of the crt suffix seems important to debian as well as my original certificate has the cer suffix. The code I added to the DOCKERFILE before the restore step is as follows:

ADD "{my-root-ca.cer}" "/temp/my-root-ca.crt"  
RUN openssl x509 -inform der -in "/temp/my-root-ca.crt" -out "/usr/local/share/ca-certificates/my-root-ca.crt"  
RUN update-ca-certificates

Where {my-root-ca.cer} is the path and filename of your corporate root CA cert. In my case I am using ADD rather than COPY because I neatened things up to use the certificate from a url webstore but it works just the same with a local copy of the cert.

FYI @noeltupas

JonDouglas commented 1 year ago

Hi friends,

I've added an initial proposal that includes functionality to disable/ignore HTTPS certificate validation. Please feel free to read through it and provide your feedback & upvote if you believe it is headed in the right direction.

https://github.com/NuGet/Home/pull/12542

KarenArzumanyan commented 1 year ago

This would be a much needed option. Especially important for CI/CD For example, when using docker-compose build inside dron-agent (docker into docker).

robindegen commented 1 year ago

With the plans to drop http, I feel it's extra important to allow validation skipping. We have internal facing servers and I don't want to pay for proper certificates, and if I'm going to self sign I might as well just disable the check since all security is out the window already. Perhaps dropping http should be reconsidered. Not everything is internet facing. Perhaps simply support and allow http from known internal ips 192.168, 10.x etc. Enforcing https in this case is a major hassle with no security benefits.

Kalyxt commented 7 months ago

Just run into this issue, I would like to host my own packages on gitea with self-signed cert but there is no workaround currently or I overlooked something ?

Nigusu-Allehu commented 6 months ago

We have added disableTLSCertificateValidation attribute that can be added to a nuget source in a NuGet config file which would allow access to sources with an invalid TLS certificate as follows

<!-- Disables certification validation on a specific https source -->
<packageSources>
    <add key="Contoso" value="https://contoso.com/packages/" disableTLSCertificateValidation="true" />
</packageSources>

WeihanLi commented 6 months ago

@Nigusu-Allehu which NuGet client version or dotnet sdk version could we expect? seemed not available now

Nigusu-Allehu commented 6 months ago

NuGet 6.11 and dotnet 8.0.400

tmds commented 6 months ago

disableTLSCertificateValidation="true"

On Linux (your machine/CI/...), a secure alternative is getting the CA certificate in a directory and setting the SSL_CERT_DIR envvar to include the system certificates and your CA directory, something like: SSL_CERT_DIR=/etc/ssl/certs:/dir_with_your_ca_cert.

NuGet / Home