Leaked UnobservedTaskExceptions AndroidMessageHandler

[rgroenewoudt]

Android application type

Android for .NET (net6.0-android, etc.)

Affected platform version

VS2022 17.4.4

Description

In our .NET 6 Android application we often see unobserved leaked exceptions (TaskScheduler.UnobservedTaskException) from AndroidMessageHander:

UnobservedException: Java.Net.SocketException: Socket closed
       at Java.Interop.JniEnvironment.InstanceMethods.CallIntMethod(JniObjectReference , JniMethodInfo , JniArgumentValue* )
       at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualInt32Method(String , IJavaPeerable , JniArgumentValue* )
       at Java.Net.HttpURLConnection.get_ResponseCode()/
       at Xamarin.Android.Net.AndroidMessageHandler.<>c__DisplayClass128_0.<DoProcessRequest>b__2()

The guilty line is https://github.com/xamarin/xamarin-android/blob/d78d7866b670324e2af0022bf7ee5531a017d211/src/Mono.Android/Xamarin.Android.Net/AndroidMessageHandler.cs#L582 If the Task.Run is not awaited anymore, its exception will end up being leaked.

Generally not a problem but it pollutes the log. Similar issues in dotnet/runtime get fixed

Steps to Reproduce

N/A

Did you find any workaround?

N/A

Relevant log output

No response

[grendello]

@rgroenewoudt it's interesting that you get Java.Net.SocketException, since we wrap the Java exception in a WebException, as we cannot swallow or handle the exception there - we still need to throw the wrapped one. As an aside, the correct thing for us to do would be to catch Java.Net.SocketException and wrap it in WebException with its status set to WebExceptionStatus.ConnectionClosed (and other possible conditions, of course), but alas, the Java exception class doesn't have any members to specify the actual cause of the exception. The only thing it gives us is the exception message which says that the connection was closed (or some other thing happened), and we can't depend on parsing that :(

To avoid the leaked exception, the connection task must be observed either in HttpClient or in the end user's code.

@simonrozsival would you mind checking if there can be any changes made to HttpClient when it calls SendAsync on our message handler to handle this particular exception?

[simonrozsival]

@grendello Hmm. I'm trying to understand what's exactly going on and it's unclear to me how the exception escapes from AndroidMessageHandler. We await all tasks in AndroidMessageHandler and the HttpClient awaits all calls to SendAsync and all of these are wrapped in try-catch blocks and all exceptions should be observed both in AndroidMessageHandler and in HttpClient. I don't think we should change HttpClient to catch this particular exception.

It would be great if we had a minimal repro of the bug. It might be quite hard to reliably reproduce though.

[grendello]

@simonrozsival yeah, I'm also puzzled about how it escapes - it should have been caught by the catch (Java.IO.Exception) block in DoSendAsync. With regards to potential HttpClient, I wondered if it made sense for it to handle the "socket closed" condition in some special way, but tbh I don't see a way to do it (considering that we don't know, in the code, what caused the Java socket exception to be thrown).

[grendello]

@rgroenewoudt would you be able to distill your code into a simple repro project so that we can check locally what's going on?

[wellhat]

We've been investigating this same error in our Xamarin Android project where uploading files above a certain size reliably reproduces the Java.Net.SocketException error (on the majority of upload attempts), while the same file upload to same CDN will always succeed on the Xamarin iOS version of the app. It actually seems to happen about at the time the upload should be completing (eg about 5s the error occurs after starting upload at 8MB/s for the 40MB file I use to reproduce this).

I can't say if I will be able to distill it down to a simple repro project but our stacktrace we see is more detailed than the one above so I'll share it anyway in case it helps with this. RetryHandler inherits from AndroidClientHandler and retries 3 times if System.Exception is encountered in SendAsync() (it's a crude implementation of retry just trying to get a workaround for this working):

ApiService.cs:66 Sending file 20221225_120824.mp4
ApiService.cs:70 Method: POST, RequestUri: 'https://web3portal.com/skynet/skyfile', Version: 2.0, Content: System.Net.Http.MultipartFormDataContent, Headers:
{
  Content-Type: multipart/form-data; boundary="2ab02153-9cc5-406f-abcd-f77c492245a5"
}
RetryHandler.cs:29 RetryHandler Sending request 
RetryHandler.cs:30 Method: POST, RequestUri: 'https://web3portal.com/skynet/skyfile', Version: 2.0, Content: System.Net.Http.MultipartFormDataContent, Headers:
{
  Skynet-Api-Key: snip
  Accept-Encoding: gzip
  Content-Type: multipart/form-data; boundary="2ab02153-9cc5-406f-abcd-f77c492245a5"
}
RetryHandler.cs:35 Trying upload: try number 1
[ViewRootImpl@f9e86ad[DropView]] ViewPostIme pointer 1
[CompatibilityChangeReporter] Compat change id reported: 160794467; UID 10679; state: DISABLED
[monodroid-assembly] open_from_bundles: failed to load assembly Windows.dll
[TrafficStats] tagSocket(127) with statsTag=0xffffffff, statsUid=-1
[OpenGLRenderer] setSurface called with nullptr
Thread started: <Thread Pool> #14
Thread started: <Thread Pool> #15
[to.hns.skydrop] Explicit concurrent copying GC freed 7705(1140KB) AllocSpace objects, 0(0B) LOS objects, 57% free, 4524KB/10MB, paused 34us,26us total 12.221ms
[to.hns.skydrop] Explicit concurrent copying GC freed 2805(2680KB) AllocSpace objects, 0(0B) LOS objects, 57% free, 4515KB/10MB, paused 24us,21us total 8.928ms
Thread started: <Thread Pool> #16
Thread started: <Thread Pool> #17
[to.hns.skydrop] Explicit concurrent copying GC freed 3215(2968KB) AllocSpace objects, 0(0B) LOS objects, 57% free, 4550KB/10MB, paused 44us,31us total 14.956ms
RetryHandler.cs:44 [ERROR] Error trying request try number 1
[monodroid] Not wrapping exception of type Java.Net.SocketException from method `SendAsync`. This will change in a future release.
RetryHandler.cs:45 [ERROR] Encoutered exception no# 1
RetryHandler.cs:45 Java.Net.SocketException: Socket closed
RetryHandler.cs:45   at Java.Interop.JniEnvironment+InstanceMethods.CallIntMethod (Java.Interop.JniObjectReference instance, Java.Interop.JniMethodInfo method, Java.Interop.JniArgumentValue* args) [0x00068] in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/obj/Release/JniEnvironment.g.cs:11725 
  at Java.Interop.JniPeerMembers+JniInstanceMethods.InvokeVirtualInt32Method (System.String encodedMember, Java.Interop.IJavaPeerable self, Java.Interop.JniArgumentValue* parameters) [0x00036] in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods_Invoke.cs:511 
  at Java.Net.HttpURLConnection.get_ResponseCode () [0x00000] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/obj/Release/monoandroid10/android-30/mcw/Java.Net.HttpURLConnection.cs:511 
  at Xamarin.Android.Net.AndroidClientHandler+<>c__DisplayClass46_0.<DoProcessRequest>b__2 () [0x00000] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:438 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:534 
  at System.Threading.Tasks.Task.Execute () [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2319 
--- End of stack trace from previous location where exception was thrown ---

  at Xamarin.Android.Net.AndroidClientHandler.DoProcessRequest (System.Net.Http.HttpRequestMessage request, Java.Net.URL javaUrl, Java.Net.HttpURLConnection httpConnection, System.Threading.CancellationToken cancellationToken, Xamarin.Android.Net.AndroidClientHandler+RequestRedirectionState redirectState) [0x00328] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:438 
  at Xamarin.Android.Net.AndroidClientHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00286] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:287 
  at SkyDrop.Droid.Helper.RetryHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000b9] in /Users/jh/Documents/source/SkyDrop/src/SkyDrop.Droid/Helper/RetryHandler.cs:40 
  --- End of managed Java.Net.SocketException stack trace ---
java.net.SocketException: Socket closed
    at java.net.SocketInputStream.read(SocketInputStream.java:188)
    at java.net.SocketInputStream.read(SocketInputStream.java:143)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readFromSocket(ConscryptEngineSocket.java:945)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:909)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readUntilDataAvailable(ConscryptEngineSocket.java:824)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.read(ConscryptEngineSocket.java:797)
    at com.android.okhttp.okio.Okio$2.read(Okio.java:138)
    at com.android.okhttp.okio.AsyncTimeout$2.read(AsyncTimeout.java:213)
    at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:307)
    at com.android.okhttp.okio.RealBufferedSource.indexOf(RealBufferedSource.java:301)
    at com.android.okhttp.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:197)
    at com.android.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:188)
    at com.android.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:129)
    at com.android.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:750)
    at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:622)
    at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:475)
    at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:411)
    at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:542)
    at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:106)
    at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:30)

RetryHandler.cs:35 Trying upload: try number 2
[TrafficStats] tagSocket(108) with statsTag=0xffffffff, statsUid=-1
RetryHandler.cs:44 [ERROR] Error trying request try number 2
RetryHandler.cs:45 [ERROR] Encoutered exception no# 2
RetryHandler.cs:45 System.ObjectDisposedException: Cannot access a closed Stream.
RetryHandler.cs:45   at System.IO.MemoryStream.EnsureNotClosed () [0x00008] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/IO/MemoryStream.cs:121 
  at System.IO.MemoryStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x0004e] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/IO/MemoryStream.cs:369 
  at System.IO.MemoryStream.ReadAsync (System.Memory`1[T] buffer, System.Threading.CancellationToken cancellationToken) [0x00016] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/IO/MemoryStream.cs:472 
--- End of stack trace from previous location where exception was thrown ---

  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.MultipartContent+ContentReadStream.ReadAsyncPrivate (System.Memory`1[T] buffer, System.Threading.CancellationToken cancellationToken) [0x0005c] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/MultipartContent.cs:476 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.IO.Stream.CopyToAsyncInternal (System.IO.Stream destination, System.Int32 bufferSize, System.Threading.CancellationToken cancellationToken) [0x0005e] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/shared/System/IO/Stream.cs:150 
  at Xamarin.Android.Net.AndroidClientHandler.WriteRequestContentToOutput (System.Net.Http.HttpRequestMessage request, Java.Net.HttpURLConnection httpConnection, System.Threading.CancellationToken cancellationToken) [0x000bb] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:364 
  at Xamarin.Android.Net.AndroidClientHandler.DoProcessRequest (System.Net.Http.HttpRequestMessage request, Java.Net.URL javaUrl, Java.Net.HttpURLConnection httpConnection, System.Threading.CancellationToken cancellationToken, Xamarin.Android.Net.AndroidClientHandler+RequestRedirectionState redirectState) [0x002aa] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:436 
  at Xamarin.Android.Net.AndroidClientHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00286] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:287 
  at SkyDrop.Droid.Helper.RetryHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000b9] in /Users/jh/Documents/source/SkyDrop/src/SkyDrop.Droid/Helper/RetryHandler.cs:40 
RetryHandler.cs:35 Trying upload: try number 3
[TrafficStats] tagSocket(127) with statsTag=0xffffffff, statsUid=-1
RetryHandler.cs:44 [ERROR] Error trying request try number 3
RetryHandler.cs:45 [ERROR] Encoutered exception no# 3
RetryHandler.cs:45 System.ObjectDisposedException: Cannot access a closed Stream.
RetryHandler.cs:45   at System.IO.MemoryStream.EnsureNotClosed () [0x00008] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/IO/MemoryStream.cs:121 
  at System.IO.MemoryStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x0004e] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/IO/MemoryStream.cs:369 
  at System.IO.MemoryStream.ReadAsync (System.Memory`1[T] buffer, System.Threading.CancellationToken cancellationToken) [0x00016] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/IO/MemoryStream.cs:472 
--- End of stack trace from previous location where exception was thrown ---

  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.MultipartContent+ContentReadStream.ReadAsyncPrivate (System.Memory`1[T] buffer, System.Threading.CancellationToken cancellationToken) [0x0005c] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/MultipartContent.cs:476 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.IO.Stream.CopyToAsyncInternal (System.IO.Stream destination, System.Int32 bufferSize, System.Threading.CancellationToken cancellationToken) [0x0005e] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/shared/System/IO/Stream.cs:150 
  at Xamarin.Android.Net.AndroidClientHandler.WriteRequestContentToOutput (System.Net.Http.HttpRequestMessage request, Java.Net.HttpURLConnection httpConnection, System.Threading.CancellationToken cancellationToken) [0x000bb] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:364 
  at Xamarin.Android.Net.AndroidClientHandler.DoProcessRequest (System.Net.Http.HttpRequestMessage request, Java.Net.URL javaUrl, Java.Net.HttpURLConnection httpConnection, System.Threading.CancellationToken cancellationToken, Xamarin.Android.Net.AndroidClientHandler+RequestRedirectionState redirectState) [0x002aa] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:436 
  at Xamarin.Android.Net.AndroidClientHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00286] in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidClientHandler.Legacy.cs:287 
  at SkyDrop.Droid.Helper.RetryHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000b9] in /Users/jh/Documents/source/SkyDrop/src/SkyDrop.Droid/Helper/RetryHandler.cs:40 
DropViewModel.cs:385 [ERROR] Encoutered exception no# 4
DropViewModel.cs:385 System.InvalidOperationException: Handler did not return a response message.
DropViewModel.cs:385   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered (System.Threading.Tasks.Task`1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) [0x000b3] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:531 
  at SkyDrop.Core.Services.ApiService.UploadFile (SkyDrop.Core.DataModels.SkyFile skyfile, System.Threading.CancellationToken ct) [0x00186] in /Users/jh/Documents/source/SkyDrop/src/SkyDrop.Core/Services/ApiService.cs:74 
  at SkyDrop.Core.ViewModels.Main.DropViewModel.UploadFile () [0x00044] in /Users/jh/Documents/source/SkyDrop/src/SkyDrop.Core/ViewModels/DropViewModel.cs:527 
  at SkyDrop.Core.ViewModels.Main.DropViewModel.FinishSendFile () [0x00236] in /Users/jh/Documents/source/SkyDrop/src/SkyDrop.Core/ViewModels/DropViewModel.cs:314 

So first a Java.Net.SocketException then the retry attempts with a new cancellation token fail with System.ObjectDisposedException: Cannot access a closed Stream.

The project is SkyDrop.Droid and the branch I ran this off is https://github.com/SkyLabs-Innovation-Group/SkyDrop/tree/feature/portal-retry-upload

[grendello]

@wellhat I'm afraid the cause of the socket exception you reported is beyond us to fix it. This issue is about the Java exception "leaking" to the caller, which we should and can fix (in fact, it shouldn't be happening even now, since we have all the exception wrappers in place, and yet...). The cause of the exception you show (including the object disposed exception, in most cases) is entirely on the Android side. If your app used .NET as opposed to "classic" Xamarin.Android, you might have had better luck using the BCL HttpClient as-is, since it implements modern TLS support based on Java interfaces, but tailored for use with the "standard" HttpClient. If you encounter issues there, we at least can do something about them, unlike with bugs/issues inside the Java API AndroidClientHandler uses. However, seeing as your app is a classic Xamarin.App one, perhaps give BTLS a try? That way you'd be able to use the standard HttpClient with, hopefully, less limitations that the wrapped Java API has.

Some historical perspective:

AndroidClientHandler was born out of necessity, because back then Mono didn't have a "native" implementation of TLS 1.2, and the only way to support it in one's app was to enable the use of Mono's BTLS (BoringSSL) interface and library which increased the APK size by a few megabytes. So, the idea was born to take advantage of the Java interfaces in Android and AndroidClientHandler was implemented as a wrapper around java.net.HttpURLConnection and friends. In the course of development, we discovered many quirks and imperfections in the Java client and a lot of complexity in AndroidClientHandler stems from the need to work around these issues (like the need to destroy the Java client instance in certain cases which, eventually, may lead to the object disposed exception). However, in the end, AndroidClientHandler (and the .NET version of it, AndroidMessageHandler) is a thin wrapper around the Java API, and an adapter between its interface and the interface/semantics required by .NET's HttpClient which frequently limits our movements and options.

[rgroenewoudt]

I have not been able to reproduce this in a small project. It is possible it is a re-thrown exception from somewhere else, like the WCF-layer.

[wellhat]

Thank you @grendello that was very helpful in that it filled some context gaps. I had a hunch that this issue was more to do with the leaking exception rather than the root cause of the Java error so thanks again for helping.

We already had btls set, and had previously tried inheriting from HttpClientHandler instead of AndroidClientHandler as a workaround, unfortunately to no avail at first; in part because we are also having this strange issue where our CancellationTokenSource on the first attempt is mysteriously getting Cancelled despite this behaviour having no possible cause and seeming pretty inexplicable (we have a cancel upload button but this was not being triggered, we commented out that only place where CancellationTokenSource.Cancel() was called, and of course the HttpClientHandler only has the token not the source so couldn't be cancelling the token either...).

That issue meant that our RetryHandler was retrying using the already cancelled token, which would immediately cancel the retries as well. The error log for the initial error before retrying, in case you're curious:

ApiService.cs:66 Sending file 20221225_120824.mp4
ApiService.cs:70 Method: POST, RequestUri: 'https://web3portal.com/skynet/skyfile', Version: 2.0, Content: System.Net.Http.MultipartFormDataContent, Headers:
{
  Content-Type: multipart/form-data; boundary="824364cd-69d2-424c-b722-e91107246a56"
}
RetryHandler.cs:29 RetryHandler Sending request 
RetryHandler.cs:30 Method: POST, RequestUri: 'https://web3portal.com/skynet/skyfile', Version: 2.0, Content: System.Net.Http.MultipartFormDataContent, Headers:
{
  Skynet-Api-Key: snip
  Accept-Encoding: gzip
  Content-Type: multipart/form-data; boundary="824364cd-69d2-424c-b722-e91107246a56"
}
RetryHandler.cs:35 Trying upload: try number 1
RetryHandler.cs:44 [ERROR] Error trying request try number 1
RetryHandler.cs:45 [ERROR] Encoutered exception no# 1
RetryHandler.cs:45 System.Threading.Tasks.TaskCanceledException: The operation was canceled.
RetryHandler.cs:45   at System.Net.Http.HttpConnection.SendAsyncCore (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x0129b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:739 
  at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync (System.Net.Http.HttpConnection connection, System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x000e6] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:330 
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x00101] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:296 
  at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00070] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:32 
  at SkyDrop.Droid.Helper.RetryHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000b9] in /Users/jh/Documents/source/SkyDrop/src/SkyDrop.Droid/Helper/RetryHandler.cs:40 
RetryHandler.cs:45 [ERROR] Logging exception - the inner exception
RetryHandler.cs:45 System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'MobileAuthenticatedStream'.
RetryHandler.cs:45   at Mono.Net.Security.MobileAuthenticatedStream.StartOperation (Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) [0x00245] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:410 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x0001b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/Common/src/CoreLib/System/Threading/Tasks/ValueTask.cs:813 
  at System.Net.Http.HttpConnection.SendAsyncCore (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00c5c] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:551 

However, I just managed to workaround that cancellation issue: first by setting it back to the Managed HttpClient handler with btls as you suggested trying, then disposing the CTS after the failed attempt and re-creating a new one & new token to pass when retrying. The main downside of this is it still requires us to use the insecure SSL handler to avoid Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED, which we set up like so:

        private RetryHandler GetInsecureMessageHandler()
        {
            var handler = new RetryHandler();

            //accept all SSL certificates (insecure!)
            handler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => true;
            return handler;
        }

However this is OK for now for our use case and thanks again for your answer which led me to try the Managed handler again and try an extra workaround for the CTS issue. Best wishes and good luck finding the cause of the leaky java exceptions.

[rgroenewoudt]

My attempts at reproducing this failed so far. If I have more information I'll reopen this.