LetsEncrypt SSL Cert produces: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

paul-kiar commented 2 years ago

Steps to Reproduce

Have a certificate with 2 verification paths as explained here
Register that certificate on a webserver

Create an HttpWebRequest to with the webserver URL from step 2

    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(address);
    request.Accept = "application/json";
    request.Method = "GET";
    using var response = await request.GetResponseAsync().ConfigureAwait(false); // throws exception

This only happens with LetsEncrypt certificates that were signed with the expired certificate DST Root CA X3. Our SSL certificate was issued in August 2021 with the dual signature. It is not an issue for Apple iOS or iPadOS

Chrome has an issue with the certificate on older devices, but not on recent devices Viewing the certificate in windows browsers showed the valid path Viewing the certificate on old emulators showed the invalid path and failed to be trusted On devices that chrome showed the certificate as valid, Xamarin Android app still failed to trust the certificate Certificate worked until September 29th when the DST Root CA X3 certificate expired

Work Around: Renewing the certificate with LetsEncrypt Acme after Sept 30th 2021 fixed the problem

Expected Behavior

SSL Works, web request succeeds

Actual Behavior

Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Version Information

Microsoft Visual Studio Enterprise 2019 Version 16.11.2 VisualStudio.16.Release/16.11.2+31624.102 Microsoft .NET Framework Version 4.8.04084

Installed Version: Enterprise

Visual C++ 2019 00435-60000-00000-AA537 Microsoft Visual C++ 2019

ADL Tools Service Provider 1.0 This package contains services used by Data Lake tools

ASA Service Provider 1.0

ASP.NET and Web Tools 2019 16.11.75.64347 ASP.NET and Web Tools 2019

ASP.NET Web Frameworks and Tools 2019 16.11.75.64347 For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0 16.11.75.64347 Azure App Service Tools v3.0.0

Azure Data Lake Node 1.0 This package contains the Data Lake integration nodes for Server Explorer.

Azure Data Lake Tools for Visual Studio 2.6.1000.0 Microsoft Azure Data Lake Tools for Visual Studio

Azure Functions and Web Jobs Tools 16.11.75.64347 Azure Functions and Web Jobs Tools

Azure Stream Analytics Tools for Visual Studio 2.6.1000.0 Microsoft Azure Stream Analytics Tools for Visual Studio

C# Tools 3.11.0-4.21403.6+ae1fff344d46976624e68ae17164e0607ab68b10 C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools 1.10 Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Extensibility Message Bus 1.2.6 (master@34d6af2) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

Fabric.DiagnosticEvents 1.0 Fabric Diagnostic Events

IntelliCode Extension 1.0 IntelliCode Visual Studio Extension Detailed Info

Microsoft Azure HDInsight Azure Node 2.6.1000.0 HDInsight Node under Azure Node

Microsoft Azure Hive Query Language Service 2.6.1000.0 Language service for Hive query

Microsoft Azure Service Fabric Tools for Visual Studio 16.10 Microsoft Azure Service Fabric Tools for Visual Studio

Microsoft Azure Stream Analytics Language Service 2.6.1000.0 Language service for Azure Stream Analytics

Microsoft Azure Stream Analytics Node 1.0 Azure Stream Analytics Node under Azure Node

Microsoft Azure Tools for Visual Studio 2.9 Support for Azure Cloud Services projects

Microsoft Continuous Delivery Tools for Visual Studio 0.4 Simplifying the configuration of Azure DevOps pipelines from within the Visual Studio IDE.

Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft Library Manager 2.1.113+g422d40002e.RR Install client-side libraries easily to any web project

Microsoft MI-Based Debugger 1.0 Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual C++ Wizards 1.0 Microsoft Visual C++ Wizards

Microsoft Visual Studio Tools for Containers 1.2 Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.

Microsoft Visual Studio VC Package 1.0 Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio 16.10.15 (552afdf) Support for debugging Mono processes with Visual Studio.

NuGet Package Manager 5.11.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

ProjectServicesPackage Extension 1.0 ProjectServicesPackage Visual Studio Extension Detailed Info

Razor (ASP.NET Core) 16.1.0.2122504+13c05c96ea6bdbe550bd88b0bf6cdddf8cde1725 Provides languages services for ASP.NET Core Razor.

Snapshot Debugging Extension 1.0 Snapshot Debugging Visual Studio Extension Detailed Info

SQL Server Data Tools 16.0.62107.28140 Microsoft SQL Server Data Tools

Test Adapter for Boost.Test 1.0 Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test 1.0 Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory.

ToolWindowHostedEditor 1.0 Hosting json editor into a tool window

TypeScript Tools 16.0.30526.2002 TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 3.11.0-4.21403.6+ae1fff344d46976624e68ae17164e0607ab68b10 Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools 16.11.0-beta.21322.6+488cc578cafcd261d90d748d8aaa7b8b091232dc Microsoft Visual F# Tools

Visual Studio Code Debug Adapter Host Package 1.0 Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Container Tools Extensions 1.0 View, manage, and diagnose containers within Visual Studio.

Visual Studio Tools for CMake 1.0 Visual Studio Tools for CMake

Visual Studio Tools for Containers 1.0 Visual Studio Tools for Containers

Visual Studio Tools for Kubernetes 1.0 Visual Studio Tools for Kubernetes

VisualStudio.DeviceLog 1.0 Information about my package

VisualStudio.Foo 1.0 Information about my package

VisualStudio.Mac 1.0 Mac Extension for Visual Studio

Xamarin 16.11.000.174 (d16-11@e8f56f1) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 16.11.0.17 (remotes/origin/11e0001f0b17269345e80b58fb3adf1ba4efe2cd@11e0001f0) Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin Templates 16.10.5 (355b57a) Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.

Xamarin.Android SDK 11.4.0.5 (d16-11/7776c9f) Xamarin.Android Reference Assemblies and MSBuild support. Mono: c633fe9 Java.Interop: xamarin/java.interop/d16-11@48766c0 ProGuard: Guardsquare/proguard/v7.0.1@912d149 SQLite: xamarin/sqlite/3.35.4@85460d3 Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-11@683f375

Xamarin.iOS and Xamarin.Mac SDK 14.20.0.25 (3b53e529b) Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

Log File

angelru commented 2 years ago

The same issue, some solution

derekcroprecords commented 2 years ago

Same issue, but renewing the certificate with LetsEncrypt Acme did not fix the problem

derekcroprecords commented 2 years ago

Workaround that worked for us was to edit the certs fullchain.pem from winacme and manually remove the last certification.

The one that says 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:0 = Digital Signature Trust Co., CN = DST Root CA X3

SeanMollet commented 2 years ago

@derekcroprecords YOU ARE MY HERO. This is definitely the easiest/best answer for this. If you're ever near KC in the US, let me know, I'll buy you several rounds of beers.

derekcroprecords commented 2 years ago

@derekcroprecords YOU ARE MY HERO. This is definitely the easiest/best answer for this. If you're ever near KC in the US, let me know, I'll buy you several rounds of beers.

Glad to help. That was rough day. That cert caused us a lot of problems. AWS lambda functions failed. Xamarin Android. Python on Raspberry Pi.

If anyone is still having problems there is a lot of information over on LetsEncrypts community forums. https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/449

daltonks commented 2 years ago

I'm running into the same issue with LetsEncrypt on my Samsung Galaxy S20+. I refreshed my certificate and updated cert-manager, but I can't connect using SignalR. ☹️ HttpClient requests work though.

LetsEncrypt's "ISRG Root X1" root certificate has the issue.

Microsoft.AspNetCore.Http.Connections.Client.HttpConnection: Error: Failed to start connection. Error getting negotiation response from 'https://scribblebuddies.app/hub'.

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00042] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Btls/MonoBtlsContext.cs:220 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:715 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:289 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:223 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x0025c] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:310 
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore (System.IO.Stream stream, System.Net.Security.SslClientAuthenticationOptions sslOptions, System.Threading.CancellationToken cancellationToken) [0x0007b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:165 
   --- End of inner exception stack trace ---
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore (System.IO.Stream stream, System.Net.Security.SslClientAuthenticationOptions sslOptions, System.Threading.CancellationToken cancellationToken) [0x000f6] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:176 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.HttpConnectionPool.CreateConnectionAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x002d8] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:408 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync (System.Threading.Tasks.ValueTask`1[TResult] creationTask) [0x000a2] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:543 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x0003f] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:284 
  at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00070] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:32 
  at Microsoft.AspNetCore.Http.Connections.Client.Internal.AccessTokenHttpMessageHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000ff] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.Internal.LoggingHttpMessageHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00095] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered (System.Threading.Tasks.Task`1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) [0x000b3] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:531 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.NegotiateAsync (System.Uri url, System.Net.Http.HttpClient httpClient, Microsoft.Extensions.Logging.ILogger logger, System.Threading.CancellationToken cancellationToken) [0x0014a] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
[0:] System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00042] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Btls/MonoBtlsContext.cs:220 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:715 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:289 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:223 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x0025c] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:310 
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore (System.IO.Stream stream, System.Net.Security.SslClientAuthenticationOptions sslOptions, System.Threading.CancellationToken cancellationToken) [0x0007b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:165 
   --- End of inner exception stack trace ---
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore (System.IO.Stream stream, System.Net.Security.SslClientAuthenticationOptions sslOptions, System.Threading.CancellationToken cancellationToken) [0x000f6] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:176 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.HttpConnectionPool.CreateConnectionAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x002d8] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:408 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync (System.Threading.Tasks.ValueTask`1[TResult] creationTask) [0x000a2] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:543 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x0003f] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:284 
  at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00070] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:32 
  at Microsoft.AspNetCore.Http.Connections.Client.Internal.AccessTokenHttpMessageHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000ff] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.Internal.LoggingHttpMessageHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00095] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered (System.Threading.Tasks.Task`1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) [0x000b3] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:531 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.NegotiateAsync (System.Uri url, System.Net.Http.HttpClient httpClient, Microsoft.Extensions.Logging.ILogger logger, System.Threading.CancellationToken cancellationToken) [0x00257] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.GetNegotiationResponseAsync (System.Uri uri, System.Threading.CancellationToken cancellationToken) [0x00080] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.SelectAndStartTransport (Microsoft.AspNetCore.Connections.TransferFormat transferFormat, System.Threading.CancellationToken cancellationToken) [0x00180] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.StartAsyncCore (Microsoft.AspNetCore.Connections.TransferFormat transferFormat, System.Threading.CancellationToken cancellationToken) [0x0011e] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at System.Threading.Tasks.ForceAsyncAwaiter.GetResult () [0x0000c] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.StartAsync (Microsoft.AspNetCore.Connections.TransferFormat transferFormat, System.Threading.CancellationToken cancellationToken) [0x00091] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnectionFactory.ConnectAsync (System.Net.EndPoint endPoint, System.Threading.CancellationToken cancellationToken) [0x00114] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at Microsoft.AspNetCore.Http.Connections.Client.HttpConnectionFactory.ConnectAsync (System.Net.EndPoint endPoint, System.Threading.CancellationToken cancellationToken) [0x001bf] in <234f60ad06d047e7b24d4168aa9bb2c7>:0 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncCore (System.Threading.CancellationToken cancellationToken) [0x000a5] in <155e5a84392943dea24ca8776c95247e>:0 
  at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncInner (System.Threading.CancellationToken cancellationToken) [0x0019e] in <155e5a84392943dea24ca8776c95247e>:0 
  at System.Threading.Tasks.ForceAsyncAwaiter.GetResult () [0x0000c] in <155e5a84392943dea24ca8776c95247e>:0 
  at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsync (System.Threading.CancellationToken cancellationToken) [0x00091] in <155e5a84392943dea24ca8776c95247e>:0 
  at Scribble.ApiRealtime.ApiRealtimeClient.<StartNewConnection>b__26_0 () [0x00170] in C:\Users\Dalton\source\repos\Scribble\Scribble\Scribble\ApiRealtime\ApiRealtimeClient.cs:170 
  at SkiEngine.Util.TaskQueue+<>c__DisplayClass14_0.<QueueAsync>b__0 (System.Threading.Tasks.Task _) [0x00059] in C:\Users\Dalton\source\repos\Scribble\SkiEngine\SkiEngine\Util\TaskQueue.cs:74

vincentcastagna commented 2 years ago

Just met the same issue while our certificates on our API was renewed. Forcing again a renewal of the certificate does not fix the issue. This impacts ALL builds.

Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/boringssl/ssl/handshake_client.c:1132

at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00042] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Btls/MonoBtlsContext.cs:220
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:715
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:289
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:223

Note : Using AndroidClientHandlersolves the issue.

angelru commented 2 years ago

Workaround that worked for us was to edit the certs fullchain.pem from winacme and manually remove the last certification.

The one that says 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:0 = Digital Signature Trust Co., CN = DST Root CA X3

I do not understand much, do you have to put this?

daltonks commented 2 years ago

One workaround (with SignalR on Android) is to just stop using LetsEncrypt. I bought SSL certs (unfortunately), plugged them into kubernetes nginx, and removed the LetsEncrypt config from my Ingress.

uzairali001 commented 2 years ago

I'm was also having the same issue but after some research trials and errors I found out the solution well it's more like a workaround. The root certificate of Let's encrypt is expired so just remove it from the yourcert-chain.prem.

Open the file in any text editor
Locate the last -----BEGIN CERTIFICATE----- at the end of the file
Delete it from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----
Save and reload the nginx

SpongeManiac commented 2 years ago

I will give this a try. I instead manually disabled the expired CA on my android device and the API works again, however it is an annoying thing for every costumer to manually change their device when I could just remove the expired cert from the chain. Although I am just the C# monkey, cutting the root cert from a key chain doesn't sound optimal

daltonks commented 2 years ago

I've seen talk on the forums to use --preferred-chain "ISRG Root X1" if it makes sense for your devices. I'm not going to test it (partly because I'm not sure where I should put this when using cert-manager), but maybe it helps someone else 😛

https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain

uzairali001 commented 2 years ago

I will give this a try. I instead manually disabled the expired CA on my android device and the API works again, however it is an annoying thing for every costumer to manually change their device when I could just remove the expired cert from the chain. Although I am just the C# monkey, cutting the root cert from a key chain doesn't sound optimal

For me I can't disturb customers as my App is an enterprise app for thousands of customers which they use daily to complete their tasks, because of this certificate issue my app was not even login and so many customers log their complain so I was in need of a quick solution but soon I will replace LE cert with comodo.

delphikit commented 2 years ago

I'm was also having the same issue but after some research trials and errors I found out the solution well it's more like a workaround. The root certificate of Let's encrypt is expired so just remove it from the yourcert-chain.prem.

Open the file in any text editor

Locate the last -----BEGIN CERTIFICATE----- at the end of the file

Delete it from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----

Save and reload the nginx

This works great. Question: Will this be inserted again by Certbot when it renews the cert?

daltonks commented 2 years ago

This works great. Question: Will this be inserted again by Certbot when it renews the cert?

I'm pretty sure that's the case unless you change the config. --preferred-chain "ISRG Root X1" might be what you're looking for, but I haven't personally tested it.

Roshek commented 2 years ago

I've seen talk on the forums to use --preferred-chain "ISRG Root X1" if it makes sense for your devices. I'm not going to test it (partly because I'm not sure where I should put this when using cert-manager), but maybe it helps someone else 😛

https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain

That method solved it for me. It's an option for the certbot cli which is used to get/renew let's encrypt certificates. We use a dockerized version of it to automatically renew our certs. The full command for renewal looks like this certbot renew --force-renewal --preferred-chain "ISRG Root X1". This will essentially result in a cert chain that is similar to the above "delete the old DST cert from the fullchain file" method.

Naxilos commented 2 years ago

There is a client-side workaround.

On the Android you can manualy disable the certificate "Digital Signature Trust Co. - DST Root CA X3".

Go to "Settings > Security > Encryption & credentials" > Trusted credentials"
Scroll down and disable "Digital Signature Trust Co. - DST Root CA X3"

In my case it works, but as I mentioned it's a workaround, not the solution.

dimonovdd commented 2 years ago

@jonathanpeppers @grendello Hi. Do you have any ideas? Does this apply to Xamarin.Android or Mono?

awatertrevi commented 2 years ago

This seems to work for me: https://github.com/xamarin/xamarin-android/issues/4688#issuecomment-658833938

dimonovdd commented 2 years ago

This seems to work for me: #4688 (comment)

How is it different from this property?

<AndroidHttpClientHandlerType>Xamarin.Android.Net.AndroidClientHandler</AndroidHttpClientHandlerType>

jvreeker commented 2 years ago

I have the same issues, I renewed the certificates with this option --preferred-chain "ISRG Root X1". Now everything seems to work again. Only I see devices with android version <=7 having issues.

Anyone an idea how to fix this?

awatertrevi commented 2 years ago

This seems to work for me: #4688 (comment)

How is it different from this property?
<AndroidHttpClientHandlerType>Xamarin.Android.Net.AndroidClientHandler</AndroidHttpClientHandlerType>

In my case I am using RestSharp. This with the default HttpClient seems to work for me.

w0mby commented 2 years ago

There is a client-side workaround.

On the Android you can manualy disable the certificate "Digital Signature Trust Co. - DST Root CA X3".

Go to "Settings > Security > Encryption & credentials" > Trusted credentials"

Scroll down and disable "Digital Signature Trust Co. - DST Root CA X3"

In my case it works, but as I mentioned it's a workaround, not the solution.

Since we do not have any possibility to change the server certificate, that's the only working one to us for the moment. Thanks!

wedun commented 2 years ago

If you use certbot on server, you need update it to latest version https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx.html

sudo apt install snapd
sudo snap install core; sudo snap refresh core
sudo apt-get remove certbot
sudo snap install --classic certbot
certbot --version
    certbot 1.19.0
sudo certbot --nginx

On older versions renew doesn't work (you will get new cert, but it doesn't work).

cpraehaus commented 2 years ago

I have the same issues, I renewed the certificates with this option --preferred-chain "ISRG Root X1". Now everything seems to work again. Only I see devices with android version <=7 having issues.

Anyone an idea how to fix this?

A client-side work-around for Android <= 7 is to import the Letsencrypt root cert manually:

Download Letsencrypt root certificate on your Smartphone: https://letsencrypt.org/certs/isrgrootx1.pem
In Android system settings go to "Security - Credential storage" and choose "Install from SD card" (this might vary depending on your phone)
Open from download folder and select the file downloaded in step 1
Set certificate name to "ISRG Root X1"
Ensure the certificate "Internet Security Research Group / ISRG Root X1" is installed
- "Security - Credential storage" - Trusted credentials"
- Look under the "User" tab

Now your App should be able to validate your LE certs.

cpraehaus commented 2 years ago

BTW, if someone's interested:

From https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/: DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates. There’s one important exception: older Android devices that don’t trust ISRG Root X1 will continue to work with Let’s Encrypt, thanks to a special cross-sign from DST Root CA X3 that extends past that root’s expiration. This exception only works for Android.

This might explain why Xamarin apps are affected if they use Managed HTTP client and not AndroidHttpClient: in the managed case the SSL validation is not done by the Android system (which probably validates correctly due to the cross-sign mentioned above) but by the managed SSL implementation (based on BoringSSL). I think this implements its own validation logic and thus fails to validate. Which is OK, since the Android behavior actually could be called a bug 😊 too.

kmiterror commented 2 years ago

Since one of external services that we use in production caused our app to crash, we introduced this temporary workaround:

This code means that all certificates will be treated as valid, it creates a security risk This will be removed once our external service provider fixes certificate on their servers

var httpHandler = new HttpClientHandler()
{
    ServerCertificateCustomValidationCallback = delegate { return true; },
});

var httpClient = HttpClientFactory.Create(httpHandler);

estebanorellana commented 2 years ago

If you use certbot on server, you need update it to latest version https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx.html
sudo apt install snapd
sudo snap install core; sudo snap refresh core
sudo apt-get remove certbot
sudo snap install --classic certbot
certbot --version
    certbot 1.19.0
sudo certbot --nginx
On older versions renew doesn't work (you will get new cert, but it doesn't work).

This solution was the one that worked for us, but we changed the last line, instead of running sudo certbot --nginx

we use certbot renew --force-renewal --preferred-chain "ISRG Root X1"

Our apps worked properly again. Thank you

peerem commented 2 years ago

It looks like it's not just a LetsEncrypt problem. Today, we changed our certificate to DigiCert, how you can see on this URL in the browser (https://www.peerem.com/robots.txt), and still have this problem.

derekcroprecords commented 2 years ago

@peerem

It looks like it's not just a LetsEncrypt problem. Today, we changed our certificate to DigiCert, how you can see on this URL in the browser (https://www.peerem.com/robots.txt), and still have this problem.

Its still a LetsEncrypt Cert on that site. run the following at the command line to view the cert chain openssl s_client -showcerts -connect peerem.com:443

gboudreau commented 2 years ago

Its still a LetsEncrypt Cert on that site.

www.peerem.com and perem.com use different certificates. The one on www.peerem.com is not using LetsEncrypt.

@peerem: Are you sure that your Xamarin app really uses www.peerem.com, and not perem.com

peerem commented 2 years ago

Thanks for the answers.

I have selected extra the "robots.txt" because it is the only static file with the "favicon.ico". Otherwise, everything except our SignalR-URL is redirected to https://peerem.com. So please do not confuse.

But Firefox ESR 78.14 also shows me the DigiCert certificate as faulty. That's all very strange!

https://www.peerem.com/robots.txt

Peer’s Certificate issuer is not recognized.

HTTP Strict Transport Security: false HTTP Public Key Pinning: false

And yes, I am sure that the App is using www.peerem.com. These are 2 completely different machines. Everything worked until yesterday.

Btw. on iOS all runs fine, with LetsEncrypt and DigiCert.

lukeschlather commented 2 years ago

I can confirm on Android ~7 BoringSSL doesn't seem to be properly trusting the ISRG Root X1. Weirdly the web browser is fine but curl seems to fail via ADB, and our Xamarin app is also failing. We've got a number of different Android versions and I'm trying to figure out what is and isn't working.

Is there an easy way to just ship the ISRG Root X1 cert with our app and do something like this?

            var handler = new HttpClientHandler();
            var isrg_cert = new X509Certificate2("ISRG_Root_X1.cer");
            X509Store store = new X509Store("custom", StoreLocation.CurrentUser);
            store.Add(isrg_cert);
            handler.ServerCertificateCustomValidationCallback =
                (httpRequestMessage, cert, certchain, policyErrors) => {
                    return store.validate(cert, certChain, store); /* this is not correct */
                };

            HttpClient httpClient = new HttpClient(handler);

peerem commented 2 years ago

Everything is now running with DigiCert for us. We forgot to merge the "Intermediate Certificate" with the certificate. This is important for the Nginx.

SeanMollet commented 2 years ago

@lukeschlather I'm seeing the same thing. Android 6 & 7 are failing in boringssl.

I've spent all day today trying to find a way to ship the certificate. I did manage to install it on the phone, which does fix API calls, but SignalR still doesn't see it.

I also managed to add it to the handler. You need to add an ssl context to the AndroidClientHandler

var isrg_cert = new X509Certificate2("ISRG_Root_X1.cer"); var handler = new Xamarin.Android.Net.AndroidClientHandler (); if(handler.TrustedCerts==null){handler.TrustedCerts = new List();} handler.TrustedCerts.Add(isrg_cert);

var httpClient = new HttpClient(handler);

That does work for the API without installing the certificate on the phone, but it doesn't work for SignalR either, since I can't find a way to pass an HttpClient into SignalR (it looks like it uses raw SSL and talks websocket on its own).

lukeschlather commented 2 years ago

Yeah, we are now using ZeroSSL which was easier and cheaper to swap in without any changes to our ACME setup (digicert's acme setup sounds annoying) and put us on a root that exists for those clients.

So SignalR doesn't use any HttpClient? That's too bad. I also found this which sounded too simple to be true: https://github.com/dotnet/runtime/issues/23666

SeanMollet commented 2 years ago

I managed to patch in the AndroidClientHandler to SignalR, but it doesn't appear to offer streams in and out the way that SignalR needs to use them. It connects, but then times out performing the handshake.

I gave up and bought a certificate.

angelru commented 2 years ago

I'm using android version 11, it still doesn't work <AndroidHttpClientHandlerType> Xamarin.Android.Net.AndroidClientHandler </AndroidHttpClientHandlerType>

granicus422 commented 2 years ago

Same. Android 11, AndroidClientHandler, and all I get is this error.

vincentcastagna commented 2 years ago

@angelru Note that if you still give a custom handler to your HttpClient constructor, this won't use the native one despite the changes in your project properties. You need to use either HttpClient() to use your config or you specify the native handler by hand HttpClient(new AndroidClientHandler());

angelru commented 2 years ago

@angelru Note that if you still give a custom handler to your HttpClient constructor, this won't use the native one despite the changes in your project properties. You need to use either HttpClient() to use your config or you specify the native handler by hand HttpClient(new AndroidClientHandler());

I don't understand why I have to do this, how do I do it with the refit?

jordimasmi commented 2 years ago

@angelru , I've done it like this, and it seems to work !

var httpClientHandler = new HttpClientHandler();

httpClientHandler.ServerCertificateCustomValidationCallback = (message, certificate, chain, sslPolicyErrors) => true; }

var refitApiClient = RestService.For( new HttpClient(httpClientHandler) { BaseAddress = new Uri(myConfig.ApiUri) } );

angelru commented 2 years ago

@angelru , I've done it like this, and it seems to work !

var httpClientHandler = new HttpClientHandler();

httpClientHandler.ServerCertificateCustomValidationCallback = (message, certificate, chain, sslPolicyErrors) => true; }

var refitApiClient = RestService.For( new HttpClient(httpClientHandler) { BaseAddress = new Uri(myConfig.ApiUri) } );

but is this not a bad practice to accept all certificates?

jordimasmi commented 2 years ago

True, but while the backend certificate is solved, you do not leave the app out of service. Obviously, it is best to have the backend certificate without issues. "Weekend fix" ... I don't want to mess up the issue thread.

angelru commented 2 years ago

How is it possible that a call from flutter works and xamarin doesn't? this makes me rethink many things ...

agusibrahim commented 2 years ago

@angelru Note that if you still give a custom handler to your HttpClient constructor, this won't use the native one despite the changes in your project properties. You need to use either HttpClient() to use your config or you specify the native handler by hand HttpClient(new AndroidClientHandler());

I don't understand why I have to do this, how do I do it with the refit?

i'm using Refit too. i make dependency handler for Xamarin.Android.Net.AndroidClientHandler

IMyOwnNetService.cs

public interface IMyOwnNetService {
    HttpClientHandler GetHttpClientHandler();
}

MyOwnNetService.cs (android)

[assembly: Dependency(typeof(MyOwnNetService))]
public class MyOwnNetService: IMyOwnNetService {
    public HttpClientHandler GetHttpClientHandler() {
        return new Xamarin.Android.Net.AndroidClientHandler();
    }
}

MyOwnNetService.cs (ios)

[assembly: Dependency(typeof(MyOwnNetService))]
public class MyOwnNetService: IMyOwnNetService {
    public HttpClientHandler GetHttpClientHandler() {
        return new HttpClientHandler();
    }
}

var hc = new HttpClient(Xamarin.Forms.DependencyService.Get<IMyOwnNetService>().GetHttpClientHandler()) {
    BaseAddress = new Uri("YOUR BASE URL")  
};
var api=RestService.For<MyAPIService>(hc);

its working for me

angelru commented 2 years ago

@angelru Note that if you still give a custom handler to your HttpClient constructor, this won't use the native one despite the changes in your project properties. You need to use either HttpClient() to use your config or you specify the native handler by hand HttpClient(new AndroidClientHandler());

I don't understand why I have to do this, how do I do it with the refit?

i'm using Refit too. i make dependency handler for Xamarin.Android.Net.AndroidClientHandler

IMyOwnNetService.cs
public interface IMyOwnNetService {
    HttpClientHandler GetHttpClientHandler();
}
MyOwnNetService.cs (android)
[assembly: Dependency(typeof(MyOwnNetService))]
public class MyOwnNetService: IMyOwnNetService {
    public HttpClientHandler GetHttpClientHandler() {
        return new Xamarin.Android.Net.AndroidClientHandler();
    }
}
MyOwnNetService.cs (ios)
[assembly: Dependency(typeof(MyOwnNetService))]
public class MyOwnNetService: IMyOwnNetService {
    public HttpClientHandler GetHttpClientHandler() {
        return new HttpClientHandler();
    }
}
var hc = new HttpClient(Xamarin.Forms.DependencyService.Get<IMyOwnNetService>().GetHttpClientHandler()) {
    BaseAddress = new Uri("YOUR BASE URL")  
};
var api=RestService.For<MyAPIService>(hc);
its working for me

for me too! thanks!!

grendello commented 2 years ago

@jonathanpeppers @grendello Hi. Do you have any ideas? Does this apply to Xamarin.Android or Mono?

It's Mono, Xamarin.Android doesn't have any code to deal with SSL, we use whatever mono/dotnet (and Android, if you use our HTTP client handler) provide.

steveisok commented 2 years ago

As @cpraehaus pointed out, the managed HttpClient has its own validation that is separate from what Android does natively. Unfortunately, that is not something we're going to be able to change.

censored commented 2 years ago

I just wanted to mention, doing the certbot renew --force-renewal --preferred-chain "ISRG Root X1" did indeed solve the issue for us.

It was an impactful hit to our GKE but it was worth it to break it for everyone for a few minutes while the cert was pulled and reissued with the chain reference to the DST cert removed.

Before

CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
---
Certificate chain
 0 s:/CN=xxxxxxxxxxxxx.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

After

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = xxxxxxxxxxxxx.com
verify return:1
---
Certificate chain
 0 s:CN = xxxxxxxxxxxxx.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

Other tidtibs folks might find useful:

Some versions of curl also can't handle this (this was version curl 7.54.0)

* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

And here's a thread at Let's Encrypt with lots of other information for those being impacted: https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired/160797

dotnet / android