SendAllAsync(IEnumerable<Message> messages, CancellationToken cancellationToken) has error with System.Net.Http.HttpRequestException: The SSL connection could not be established

[bingtianyiyan]

[REQUIRED] Step 2: Describe your environment

• Operating System version: Linux
• Firebase SDK version: 1.170
• Firebase Product: _____ (auth, database, storage, etc)
• .NET version: 3.1
• OS: _____

Relevant Code:

coeds: and i use polly to retry once when error ocur

await FirebaseMessaging.DefaultInstance.SendMulticastAsync(message)

// TODO(you): code here to reproduce the problem job execute every one minutes once,we have 150000 batch count ,and one minutes with 9000 batch count, the ocur happen when we send msg one minutes total count is 9000 batch count,with 90 batch count once send it ,but somestime it will happen errors with below:

FirebaseAdmin.Messaging.FirebaseMessagingException: Unknown error while making a remote service call: The SSL connection could not be established, see inner exception. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken) at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState) at System.Net.Security.SslStream.<>c.b65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state) at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1,TArg2](Func5 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions) at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Google.Apis.Requests.BatchRequest.ExecuteAsync(CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendBatchRequestAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) --- End of inner exception stack trace --- at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendAllAsync(IEnumerable`1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message) at Job.DomainService.FirebaseMgr.FirebaseSendMsgHelper.<>c__DisplayClass0_0.<b1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy.ExecuteAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken) at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState) at System.Net.Security.SslStream.<>c.b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state) at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1,TArg2](Func5 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions) at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Google.Apis.Requests.BatchRequest.ExecuteAsync(CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendBatchRequestAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable`1 messages, Boolean dryRun, CancellationToken cancellationToken)

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[bingtianyiyan]

I think it is caused by too frequent request interface, but there is no clear official limit, we have purchased an unlimited service

[bingtianyiyan]

Now I Task.delay(100) 100 milliseconds before sending the call, and unfortunately I still get the same error,so i take to issue,and i see the source code and no find any solutions

[hiranya911]

---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)

Looks like a network issue or some other low-level SSL handshake problem. Does it happen for all invocations or only some of them?

I came across the following discussion thread which seems to outline a similar problem: https://stackoverflow.com/questions/30664566/authentication-failed-because-remote-party-has-closed-the-transport-stream

[bingtianyiyan]

it happen in sometime ,It's okay to visit less frequently,However, when a large amount of data is sent in batches, the above errors may start to be reported within a certain 10 minutes, and then stop when the data is almost ready to be sent.
Yesterday, there were about 15W pieces of data, 90 pieces for each batch. It was pushed within about 30 minutes after production, and this kind of problem was reported in about 10 minutes.The total number of visits was 1666 low-level SSL handshake problem?happen in server end or push end ?
y

[bingtianyiyan]

today:also have blow questions ,and have new questions with sdk-version=2.0.0.0 ,i update nuget to 2.0.0.0,and batch send=500,and error is low then yesterday , batch num=100 . I think requently use api in small time will happen。

System.Threading.Tasks.TaskCanceledException: A task was canceled. at Google.Apis.Auth.OAuth2.TokenRefreshManager.GetAccessTokenForRequestAsync(CancellationToken cancellationToken) at Google.Apis.Auth.OAuth2.ServiceAccountCredential.GetAccessTokenForRequestAsync(String authUri, CancellationToken cancellationToken) at Google.Apis.Auth.OAuth2.ServiceCredential.GetAccessTokenWithHeadersForRequestAsync(String authUri, CancellationToken cancellationToken) at Google.Apis.Auth.OAuth2.ServiceCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Google.Apis.Http.ConfigurableMessageHandler.CredentialInterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Google.Apis.Requests.BatchRequest.ExecuteAsync(CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendBatchRequestAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendAllAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message)

--- End of stack trace from previous location where exception was thrown --- at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy.ExecuteAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)

[bingtianyiyan]

i download google.api.core github code source:chage this add code below -> handler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }; but is should see result in product environment tomorrow

///

/// Creates a simple client handler with redirection and compression disabled. ///

private HttpClientHandler CreateAndConfigureClientHandler() { var handler = CreateClientHandler(); if (handler.SupportsRedirectConfiguration) { handler.AllowAutoRedirect = false; } if (handler.SupportsAutomaticDecompression) { handler.AutomaticDecompression = DecompressionMethods.None; } //Add this handler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }; return handler; }

[hiranya911]

@bingtianyiyan I'm having some difficulty understanding your implementation and the steps you've tried so far. It would be great if you can describe your problem with some code (ideally a complete, minimal example that others can run to reproduce the error).

System.Threading.Tasks.TaskCanceledException: A task was canceled.

That looks like something explicitly cancelled the send operation. My guess is Polly cancelled a pending operation, and retried (I'm not familiar with Polly, so I cannot confirm).

[bingtianyiyan]

it occur in sometimes in data is to frequently send,not always,One or two calls are not visible, I suggest to do a large amount of data batch send pressure test。i use below code to send and use polly to retry if fail。you point problem：System.Threading.Tasks.TaskCanceledException: A task was canceled you point is version2.0.0 sdk inner code have refresh token before send msg to check token is expired and get refresh token,but sometimes fail and cancel task when if slow or something occur。polly use when fail will retry ,It does not cancel the running task。 i this use task to refresh token is not good method,like use cache or distribute cache sdk:Firebase SDK version: 2.0.0.0 code below：

parmas message: var message = new MulticastMessage() { Data = new Dictionary<string, string> { { "Title","test"}, { "Body","test"}, { "Type","1"} }, Tokens = nes string[] {"you token"} }; ///

/// ///

/// public static async Task SendMessageBatchAsync(MulticastMessage message) { return await Policy.Handle() .WaitAndRetryAsync(1, retryAttempt => TimeSpan.FromSeconds(Math.Pow(20, retryAttempt))) .ExecuteAsync(async () => { return await FirebaseMessaging.DefaultInstance.SendMulticastAsync(message); } ); }

[bingtianyiyan]

today product works better then yesterday,and only have single one error occur -> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. I find the microsoft issue have people take->url https://docs.microsoft.com/en-us/answers/questions/25701/...hentication-fai.html
below url is Chinese enginer find the solutions,but i also need to try. https://www.cnblogs.com/leoxjy/p/11235028.html

so i think is https and have Certificate,we should do special hander.

so i will add additional code below like this.and wait tomorrow the result of running batch data。I have no better choice to retry error to occur like product. ///

/// Creates a simple client handler with redirection and compression disabled. ///

private HttpClientHandler CreateAndConfigureClientHandler() { var handler = CreateClientHandler(); if (handler.SupportsRedirectConfiguration) { handler.AllowAutoRedirect = false; } if (handler.SupportsAutomaticDecompression) { handler.AutomaticDecompression = DecompressionMethods.None; } // add code handler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; };

       //today add code 
       handler.SslProtocols = SslProtocols.Ssl3 | SslProtocols.Tls12 | SslProtocols.Tls11 | SslProtocols.Tls;
        handler.ClientCertificateOptions = ClientCertificateOption.Manual;
        return handler;
    }

[hiranya911]

I don't fully understand what you're are trying to tell. But I will try my best to explain a few things that might be helpful.

1. Refreshing and fetching tokens is done by the Google.Apis.Auth package. As far as I know, the behavior of this package has not changed in a long time.
2. The refresh tokens task runs as part of the same async task as your send operation. It is not a separate background task.
3. FirebaseAdmin never cancels any tasks. I doubt Google.Apis.Auth does either. So my guess is it's Polly that's cancelling tasks in order to retry them.
4. It seems you can register a pre-retry callback with Polly. I suggest you implement that and log all the information for debug purposes:

Policy
  .Handle<SomeExceptionType>()
  .WaitAndRetry(new[]
  {
    TimeSpan.FromSeconds(1),
    TimeSpan.FromSeconds(2),
    TimeSpan.FromSeconds(3)
  }, (exception, timeSpan, context) => {
    // Add logic to be executed before each retry, such as logging    
  });

For the most part this issue looks like an issue caused by the runtime environment (TLS config or other dependencies) or the network. It's unlikely to be a general bug. I'd recommend reaching out to Firebase Support for further help: https://firebase.google.com/support/troubleshooter/report

[bingtianyiyan]

Thanks for your advice。 today works fine! The network engineer added it to Docker pods, and it's already been added to my code.。

[hiranya911]

I'm going to close this since I don't see a reason to believe this is the direct result of a bug in the codebase. Feel free to contact Firebase Support if the problem continues to occur.