dotnet-issue-labeler[bot]
commented
3 years ago I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
Closed otisaardvark closed 3 years ago
dotnet-issue-labeler[bot]
commented
3 years ago I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
mthalman
commented
3 years ago @nkolev92 - Can you help us troubleshoot this?
nkolev92
commented
3 years ago NuGet communicates via http in a very standard way. @zivkan has done some investigations and generate some guidance.
https://developercommunity2.visualstudio.com/t/VS-2019-Publish-issue---Unable-to-load-t/1257366#T-N1257595 is the best I could find, but maybe he has some better instructions?
otisaardvark
commented
3 years ago @nkolev92 - Thanks for the input.
Reading the link followed a couple of the referenced links further and found this comment
If this app shows that Tls12 worked, but None failed, the most common reason is that Windows TLS 1.3 is enabled, however is incompatible with api.nuget.org's CDN provider's implementation. You should disable TLS 1.3 so that Windows no longer attempts to use TLS 1.3
The output from the openssl test that I did contained data that suggested the call to *pkg.github.com:443 was trying to use TLS 1.3 which could be the issue.
For the Docker build to work on WIndows, I assume I would have to disable TLS1.3 support in the WIndows host.
Is that a correct assumption or would that need to be done in the docker image?
zivkan
commented
3 years ago My understanding is that newer versions of Windows 10 have a new TLS 1.3 implementation so should work now. But as Nikolche wrote, we use HttpClient in a very standard way. A console app pubic async Task Main() { await new HttpClient().GetAsString("https://api.nuget.org/v3/index.json"); } should have the same connection failure/success as NuGet does. I occasionally dig deeper when we have several customer complaints, but I'm not an expert on HttpClient's implmentation on Windows, or Windows' SSL library, or the myriad of registry settings that modify the behaviour. System.Net experts from dotnet/runtime would probably be more capable to troubleshoot than I am.
otisaardvark
commented
3 years ago @zikvan, that's what's confused me somewhat. A dotnet restore works perfectly from the Windows 10 but the same step in the Docker build is failing.
Further, from the detailed output, I can see that the first stage of the restore process succeeds as the package from the private repository is located from the call to https://nuget.pkg.github.com/<github_user>/index.json.
#15 8.815 1>/usr/share/dotnet/sdk/5.0.202/NuGet.targets(131,5): error : The feed 'Github_UserRepo [https://nuget.pkg.github.com/<github_user>/index.json]' lists package 'GlobalServicesTools.PagedList.1.0.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. [/src/Services/<service_name>/<service_name>.API/<service_name>.API.csproj]It seems that the call to the download link https://nuget.pkg.github.com/<github_user>/download/globalservicestools.pagedlist/1.0.0/globalservicestools.pagedlist.1.0.0.nupkg fails.
#15 8.853 /usr/share/dotnet/sdk/5.0.202/NuGet.targets(131,5): error : Failed to download package 'GlobalServicesTools.PagedList.1.0.0' from 'https://nuget.pkg.github.com/<github_user>/download/globalservicestools.pagedlist/1.0.0/globalservicestools.pagedlist.1.0.0.nupkg'. [/src/Services/<service_name>/<service_name>.API/<service_name>.API.csproj]However, if I copy this link from the Docker output and post it to my browser, the download will succeed without error although it does seem to take several seconds to start. I did note, that the link redirects to https://github-registry-files.githubusercontent.com/.
mthalman
commented
3 years ago You should be able to write a simple .NET console app that uses HttpClient to connect to https://nuget.pkg.github.com/<github_user>/download/globalservicestools.pagedlist/1.0.0/globalservicestools.pagedlist.1.0.0.nupkg and run that in the container. It'd be interesting to know the result of that.
otisaardvark
commented
3 years ago @mthalman I put togtether a really simple console app to grab the URL.
static void Main(string[] args)
{
string uri = "https://nuget.pkg.github.com/<github_user>/download/globalservicestools.pagedlist/1.0.0/globalservicestools.pagedlist.1.0.0.nupkg";
string paToken = "GITHUB_ACCESS_TOKEN";
var request = WebRequest.Create(uri) as HttpWebRequest;
request.UserAgent = "dotnet/csharp web-request";
request.Method = "GET";
request.Headers.Add(HttpRequestHeader.Authorization, "token " + paToken);
Stopwatch watch = new Stopwatch();
watch.Start();
using (WebResponse response = request.GetResponse())
{
MemoryStream stream = new MemoryStream();
response.GetResponseStream().CopyTo(stream);
byte[] data = stream.ToArray();
Console.WriteLine($"Recieved {data.Length} bytes");
}
watch.Stop();
Console.WriteLine($"Total process time = {watch.Elapsed:c}");
}When I run it from a Windows 10 CLI, I get the expected output as follows
> dotnet run
Received 10855 bytes
Total process time = 00:00:01.5754266However, when containerize it (using a similar Dockerfile) it will builds without issue. However, when I run the Docker container, I get the same error as my original problem with an SSL PartialChain error.
>docker run nuget_test
Unhandled exception. System.Net.WebException: The SSL connection could not be established, see inner exception.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: PartialChain
at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at System.Net.HttpWebRequest.SendRequest(Boolean async)
at System.Net.HttpWebRequest.GetResponse()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at NugetTest.Program.Main(String[] args) in /src/Program.cs:line 23For reference, this is the Dockerfile I used -- it may be overkill, but I kept it similar to the original issue. The RUN dotnet restore works fine as there are no package to restore.
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS base
WORKDIR /app
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["*.csproj", ""]
RUN dotnet restore "NugetTest.csproj"
COPY . .
RUN dotnet build "NugetTest.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "NugetTest.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "NugetTest.dll"]Ok, so this doesn't seem specific to NuGet.
@bartonjs - Are you familiar with SSL PartialChain errors and why that might be happening here?
bartonjs
commented
3 years ago The PartialChain error code means that the chain engine could not find a series of certificates from the end-entity certificate up through a self-signed (root) certificate.
Generally, for a TLS connection the server sends their certificate and all intermediates between their end-entity certificate and the root, but not the root itself -- the idea being that the client machine either doesn't trust the root or already has a copy of that certificate it can use, so it's a waste of bandwidth.
Doing a test against nuget.pkg.github.com right now, I see that it presents two certificates:
0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.pkg.github.com
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
1 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CAOn my test machine the DigiCert High Assurance EV Root CA root is known and trusted, so everything's copacetic for me.
Since I'd expect that root to also be known to, and trusted by, Windows Docker containers, it seems like the most likely explanation is that the container was running in something like a captive network environment and it got some alternate certificate that it couldn't work out.
What's interesting, though, is that DigiCert SHA2 High Assurance Server CA does not provide a URL for where to obtain its issuer certificate if you don't already have it. That means that if the container didn't already know about DigiCert High Assurance EV Root CA then it'd have no way to find it, which would result in... PartialChain. So the other possibility is that something has scoured the root trust list on the container... or it's running in a mode where it can't do the "umm... I see I need to ask WU for the updated root list" due to network restrictions.
mthalman
commented
3 years ago So this is the output I see from the Windows container when checking the roots:
❯ docker run --rm mcr.microsoft.com/dotnet/sdk:5.0 pwsh -c "dir Cert:\CurrentUser\Root | select Subject"
Subject
-------
CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com
CN=Thawte Timestamping CA, OU=Thawte Certification, O=Thawte, L=Durbanville, S…
CN=Microsoft Root Authority, OU=Microsoft Corporation, OU=Copyright (c) 1997 M…
CN=Symantec Enterprise Mobile Root for Microsoft, O=Symantec Corporation, C=US
CN=Microsoft Root Certificate Authority 2011, O=Microsoft Corporation, L=Redmo…
CN=Microsoft Authenticode(tm) Root Authority, O=MSFT, C=US
CN=Microsoft Root Certificate Authority 2010, O=Microsoft Corporation, L=Redmo…
CN=Microsoft ECC TS Root Certificate Authority 2018, O=Microsoft Corporation, …
OU=Copyright (c) 1997 Microsoft Corp., OU=Microsoft Time Stamping Service Root…
OU="NO LIABILITY ACCEPTED, (c)97 VeriSign, Inc.", OU=VeriSign Time Stamping Se…
CN=Microsoft ECC Product Root Certificate Authority 2018, O=Microsoft Corporat…
OU=Class 3 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
OU=Class 3 Public Primary Certification Authority, O="VeriSign, Inc.", C=USI don't see DigiCert High Assurance EV Root CA listed in here. @bartonjs - Is that the issue then?
bartonjs
commented
3 years ago That seems like a good piece of evidence, yeah.
mthalman
commented
3 years ago I attempted to repro this myself and was unsuccessful. I published a package to my own GitHub package repo and attempted to download it directly with the above code from @otisaardvark inside of a Windows container (mcr.microsoft.com/dotnet/sdk:5.0). It worked just fine for me.
@otisaardvark - Are you pulling down the latest version of mcr.microsoft.com/dotnet/sdk:5.0 in your testing? What version of Windows are you running (C:\>winver)?
otisaardvark
commented
3 years ago @mthalman - I am using the most recent Windows container mcr.microsoft.com/dotnet/sdk:5.0 in my Dockerfile. I deleted the base image and rebuilt the container as I noticed there had been an update (5.0.203) since I opened the issue but I still have the error.
I am using Windows 10 Pro build 19042.929
otisaardvark
commented
3 years ago @mthalman - Something by way of an update on some further testing....
I installed Docker Desktop on a different Windows 10 machine and cloned my test repo that fails on my primary development machine. Both the native and the containerized builds ran without any issue. So I am left with the thought that it must be something machine dependent that is causing the certificate issue.
What, in the host machines configuration, could cause an issue in the Docker build process? Compiling and running the app without Docker is error-free. As soon as Docker is introduced it fails?
mthalman
commented
3 years ago I'm going to convert this from an issue to a discussion since you've been able to get this working on a different machine. I agree that it seems to be a machine configuration issue but I don't know what that might be. In any case, it doesn't seem to be an issue with the configuration of the Docker image itself. If there is an issue with .NET here, it would be in the https://github.com/dotnet/runtime repo but there needs to be a reliable repro.
Describe the Bug
On WIndows 10, the
RUN dotnet restorestep fails during a docker build when the project references one or more Nuget packages hosted in a private Github repository and throw a SSL PartialChain error. The error is being thrown at the point an attempt is made to download the package but after the package has been located in the package repository.The process when run on a Linux laptop (Ubuntu 20.04) using the same Docker build files and project source code completes without any error.
The equivalent
dotnet restoreanddotnet buildrun from the Windows CLI for the same project complete without error.Steps to Reproduce
docker-compose.yml
Dockerfile
Other Information
The following error is returned by the process
The
Nuget.configfile that is copied in the build process contains sources for the public nuget package repository and the private github package repository complete with username and personal access token (for the purpose of testing only). Removing the PAT from the nuget file results in a 401 (Unauthorised) error, so it does not appear to be an issue with config.Adding the
--verbosity detailedoption to thedotnet restoreindicates that the package is correctly located in the repository, but the download action fails with the error above.I've checked the certificate chain in the docker image for the Github package repository on both Windows and Linux using the following command: -
The same output returned on both platforms that suggest the certificate chain is complete and not a partial chain as the error indicates.
Output of
docker versionOutput of
docker info