Closed bingtianyiyan closed 3 years ago
I found a few problems with this issue:
I think it is caused by too frequent request interface, but there is no clear official limit, we have purchased an unlimited service
Now I Task.delay(100) 100 milliseconds before sending the call, and unfortunately I still get the same error,so i take to issue,and i see the source code and no find any solutions
---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Looks like a network issue or some other low-level SSL handshake problem. Does it happen for all invocations or only some of them?
I came across the following discussion thread which seems to outline a similar problem: https://stackoverflow.com/questions/30664566/authentication-failed-because-remote-party-has-closed-the-transport-stream
it happen in sometime ,It's okay to visit less frequently,However, when a large amount of data is sent in batches, the above errors may start to be reported within a certain 10 minutes, and then stop when the data is almost ready to be sent.
Yesterday, there were about 15W pieces of data, 90 pieces for each batch. It was pushed within about 30 minutes after production, and this kind of problem was reported in about 10 minutes.The total number of visits was 1666
low-level SSL handshake problem?happen in server end or push end ?
y
today:also have blow questions ,and have new questions with sdk-version=2.0.0.0 ,i update nuget to 2.0.0.0,and batch send=500,and error is low then yesterday , batch num=100 . I think requently use api in small time will happen。
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Google.Apis.Auth.OAuth2.TokenRefreshManager.GetAccessTokenForRequestAsync(CancellationToken cancellationToken)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.GetAccessTokenForRequestAsync(String authUri, CancellationToken cancellationToken)
at Google.Apis.Auth.OAuth2.ServiceCredential.GetAccessTokenWithHeadersForRequestAsync(String authUri, CancellationToken cancellationToken)
at Google.Apis.Auth.OAuth2.ServiceCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Google.Apis.Http.ConfigurableMessageHandler.CredentialInterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Google.Apis.Requests.BatchRequest.ExecuteAsync(CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendBatchRequestAsync(IEnumerable
1 messages, Boolean dryRun, CancellationToken cancellationToken)
at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendAllAsync(IEnumerable
1 messages, Boolean dryRun, CancellationToken cancellationToken)
at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun, CancellationToken cancellationToken)
at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun)
at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message)
--- End of stack trace from previous location where exception was thrown ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates
1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable
1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy.ExecuteAsync[TResult](Func
3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
i download google.api.core github code source:chage this add code below -> handler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }; but is should see result in product environment tomorrow
///
@bingtianyiyan I'm having some difficulty understanding your implementation and the steps you've tried so far. It would be great if you can describe your problem with some code (ideally a complete, minimal example that others can run to reproduce the error).
System.Threading.Tasks.TaskCanceledException: A task was canceled.
That looks like something explicitly cancelled the send operation. My guess is Polly cancelled a pending operation, and retried (I'm not familiar with Polly, so I cannot confirm).
it occur in sometimes in data is to frequently send,not always,One or two calls are not visible, I suggest to do a large amount of data batch send pressure test。i use below code to send and use polly to retry if fail。you point problem:System.Threading.Tasks.TaskCanceledException: A task was canceled you point is version2.0.0 sdk inner code have refresh token before send msg to check token is expired and get refresh token,but sometimes fail and cancel task when if slow or something occur。polly use when fail will retry ,It does not cancel the running task。 i this use task to refresh token is not good method,like use cache or distribute cache sdk:Firebase SDK version: 2.0.0.0 code below:
parmas message:
var message = new MulticastMessage()
{
Data = new Dictionary<string, string>
{
{ "Title","test"},
{ "Body","test"},
{ "Type","1"}
},
Tokens = nes string[] {"you token"}
};
///
today product works better then yesterday,and only have single one error occur -> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
I find the microsoft issue have people take->url https://docs.microsoft.com/en-us/answers/questions/25701/...hentication-fai.html
below url is Chinese enginer find the solutions,but i also need to try.
https://www.cnblogs.com/leoxjy/p/11235028.html
so i think is https and have Certificate,we should do special hander.
so i will add additional code below like this.and wait tomorrow the result of running batch data。I have no better choice to retry error to occur like product.
///
//today add code
handler.SslProtocols = SslProtocols.Ssl3 | SslProtocols.Tls12 | SslProtocols.Tls11 | SslProtocols.Tls;
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
return handler;
}
I don't fully understand what you're are trying to tell. But I will try my best to explain a few things that might be helpful.
Google.Apis.Auth
package. As far as I know, the behavior of this package has not changed in a long time.FirebaseAdmin
never cancels any tasks. I doubt Google.Apis.Auth
does either. So my guess is it's Polly that's cancelling tasks in order to retry them. Policy
.Handle<SomeExceptionType>()
.WaitAndRetry(new[]
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(3)
}, (exception, timeSpan, context) => {
// Add logic to be executed before each retry, such as logging
});
For the most part this issue looks like an issue caused by the runtime environment (TLS config or other dependencies) or the network. It's unlikely to be a general bug. I'd recommend reaching out to Firebase Support for further help: https://firebase.google.com/support/troubleshooter/report
Thanks for your advice。 today works fine! The network engineer added it to Docker pods, and it's already been added to my code.。
I'm going to close this since I don't see a reason to believe this is the direct result of a bug in the codebase. Feel free to contact Firebase Support if the problem continues to occur.
[REQUIRED] Step 2: Describe your environment
Relevant Code:
coeds: and i use polly to retry once when error ocur
await FirebaseMessaging.DefaultInstance.SendMulticastAsync(message)
// TODO(you): code here to reproduce the problem job execute every one minutes once,we have 150000 batch count ,and one minutes with 9000 batch count, the ocur happen when we send msg one minutes total count is 9000 batch count,with 90 batch count once send it ,but somestime it will happen errors with below:
FirebaseAdmin.Messaging.FirebaseMessagingException: Unknown error while making a remote service call: The SSL connection could not be established, see inner exception. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken) at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState) at System.Net.Security.SslStream.<>c.b65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactoryb 1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Funcb__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory
1.FromAsyncImpl[TArg1,TArg2](Func
5 beginMethod, Func2 endFunction, Action
1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions) at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Google.Apis.Requests.BatchRequest.ExecuteAsync(CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendBatchRequestAsync(IEnumerable
1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable1 messages, Boolean dryRun, CancellationToken cancellationToken) --- End of inner exception stack trace --- at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable
1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendAllAsync(IEnumerable`1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message, Boolean dryRun) at FirebaseAdmin.Messaging.FirebaseMessaging.SendMulticastAsync(MulticastMessage message) at Job.DomainService.FirebaseMgr.FirebaseSendMsgHelper.<>c__DisplayClass0_0.<3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates
1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable
1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy.ExecuteAsync[TResult](Func
3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken) at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState) at System.Net.Security.SslStream.<>c.1.FromAsyncImpl[TArg1,TArg2](Func
5 beginMethod, Func2 endFunction, Action
1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions) at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Google.Apis.Requests.BatchRequest.ExecuteAsync(CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendBatchRequestAsync(IEnumerable
1 messages, Boolean dryRun, CancellationToken cancellationToken) at FirebaseAdmin.Messaging.FirebaseMessagingClient.SendAllAsync(IEnumerable`1 messages, Boolean dryRun, CancellationToken cancellationToken)