Open haavamoa opened 4 months ago
@simonrozsival Now that we are part of the dotnet
GitHub organization, should I transfer HTTP issues like this to dotnet/runtime
?
@jpobst this might be an issue in Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean )
so I would keep it in the android
repo for now, until we can say that this issue doesn't belong here.
I forgot to mention another work around I've had success with:
This might end up with throwing another dispose exception, or not. I've added a 5 second retry, which will end up with a success at some point during those 5 seconds.
If you are interested:
Here's my HttpClient helper class that gets run when an ObjectDisposedException
occurs in a DelegateHandler:
internal static class SharedRetryHelper
{
internal static Task<HttpResponseMessage> RetryDueToAndroidBug(this IHttpClient httpClient, HttpRequestMessage requestMessage, CancellationToken cancellationToken,ObjectDisposedException objectDisposedException)
{
var dateTimeNow = DateTime.Now;
var now = new TimeOnly(dateTimeNow.Hour, dateTimeNow.Minute, dateTimeNow.Second);
if (requestMessage.RequestUri == null) throw objectDisposedException;
if (httpClient.DisposeInformation.LastTimeDisposed != null)
{
var nowTicks = DateTime.Now.Ticks;
var lastTime = httpClient.DisposeInformation.LastTimeDisposed.Value;
if (lastTime.Add(TimeSpan.FromSeconds(5)).Ticks > nowTicks) //If it's been more than 5seconds of disposal exceptions.
{
httpClient.DisposeInformation.LastTimeDisposed = null;
Log(httpClient, requestMessage, "Got disposed exception on Android for more than 5 seconds. Will give up and throw exception.");
throw objectDisposedException;
}
}
else
{
httpClient.DisposeInformation.LastTimeDisposed = now;
}
httpClient.DisposeInformation.TotalTimesObjectDisposed++;
Log(httpClient, requestMessage, $"Got disposed exception on Android. Has been disposed {httpClient.DisposeInformation.TotalTimesObjectDisposed} times, last time was {httpClient.DisposeInformation.LastTimeDisposed.Value}:{httpClient.DisposeInformation.LastTimeDisposed.Value.Second}, will retry now at {now}:{now.Second}");
if (requestMessage.Method == HttpMethod.Get)
{
return httpClient.Get(requestMessage.RequestUri.AbsoluteUri, cancellationToken);
}
if (requestMessage.Method == HttpMethod.Post)
{
return httpClient.Post(requestMessage.RequestUri.AbsoluteUri, requestMessage.Content, cancellationToken);
}
if (requestMessage.Method == HttpMethod.Put)
{
return httpClient.Put(requestMessage.RequestUri.AbsoluteUri, requestMessage.Content, cancellationToken);
}
if (requestMessage.Method == HttpMethod.Delete)
{
return httpClient.Delete(requestMessage.RequestUri.AbsoluteUri, cancellationToken);
}
throw objectDisposedException;
}
private static void Log(IHttpClient httpClient, HttpRequestMessage requestMessage, string message)
{
#if __ANDROID__
if (requestMessage.RequestUri != null)
{
Android.Util.Log.Debug("DME HttpClientAdapter",$"HttpClientName: {httpClient.Name} : {requestMessage.RequestUri.AbsoluteUri} : {message}");
}
#endif
}
IHttpClient
is just an abstraction on top of HttpClient in our project. The DispiseInformatin is just a simple class containing when it last got the bug and a counter of how many times it happened. This is convenient for debugging purposes.
@simonrozsival: Just shout out if you need me to test potential nightly builds or something in our project.
@haavamoa can you please share more information about the settings you're using with the client? I'm especially interested in automatic decompression, built-in authentication (Basic?, NTLM?), proxies, ... Also are you able to tell what happened with the request just before it threw the exception (redirect, 4xx error, 5xx error)? Does this happen for a specific HTTP method or does it happen both with and without any body sent to the server?
Hi @simonrozsival.
We've seen this bug on both HttpClients that is using authentication, but also on HttpClients which his not. For the ones that do ; we are using built-in-authentication using Bearer Token from the Identity Model flow. We've not set any decompression or proxies to our HttpClients.
No requests have failed before we do the calls. One example is simply trying to reach a /status/ping endpoint which does nothing more than return 200 OK. This fail sporadicly. It happens for all kinds of HTTP methods, with or without bodies.
I've spent days trying to see the connection between the order of doing calls, or what happens to them before it fails, but I am unable to find a pattern at all. From my point of view it can happen to any call we do regardless of the situation, which I find too good to be true to be honest.
Thanks for the details, @haavamoa. I remember seeing reports of ObjectDisposedException
being thrown before but we've never been able to reproduce it. I will try to repro this again to see if I can narrow down what could cause this exception.
If you think of any additional details (are you making requests in parallel or just one at a time, you're making requests always when the app is in foreground or if the app is in background, ...) or if you're able to repro it reliably, please let me know.
@haavamoa could you try building your app with <UseNativeHttpHandler>false</UseNativeHttpHandler>
? That will internally change AndroidMessageHandler
for SocketsHttpHandler
and it might help you avoid this issue altogether.
Hey @simonrozsival, I have started observing this exception randomly in our http requests (MAUI) after the tooling was upgraded to Android SDK 34.0.113. Looking through our logs I do not see any discernable patterns for when the failure happens.
@zachdean do you have any stacktraces you can share? Are you able to reproduce the exception in your app?
Same happens for us after workload was updated to 34.0.113 34.0.95 works ok
As a workaround we will try this:
Make a file like this named workload.json
:
{
"microsoft.net.sdk.android": "34.0.95/8.0.100"
}
Then dotnet workload update --from-rollback-file $(androidProjectFolder)/workload.json
should install 34.0.95.
Stacktrace of the crash:
An error occured deserializing the response.
Refit.ApiException: An error occured deserializing the response.
---> System.ObjectDisposedException: ObjectDisposed_Generic
ObjectDisposed_ObjectName_Name, Java.IO.InputStreamInvoker
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable )
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String , IJavaPeerable , JniArgumentValue* )
at Java.IO.InputStream.Close()
at Android.Runtime.InputStreamInvoker.Close()
at System.IO.Stream.Dispose()
at System.IO.BufferedStream.Dispose(Boolean )
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.IO.DelegatingStream.Dispose(Boolean )
at System.IO.Stream.Close()
at System.IO.StreamReader.Dispose(Boolean )
at System.IO.StreamReader.Close()
at Newtonsoft.Json.JsonTextReader.Close()
at Newtonsoft.Json.JsonReader.Dispose(Boolean disposing)
at Newtonsoft.Json.JsonReader.System.IDisposable.Dispose()
at Refit.NewtonsoftJsonContentSerializer.<FromHttpContentAsync>d__4`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at Refit.RequestBuilderImplementation.<DeserializeContentAsync>d__15`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at Refit.RequestBuilderImplementation.<>c__DisplayClass14_0`2.<<BuildCancellableTaskFuncForMethod>b__0>d[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
Exception_EndOfInnerExceptionStack
at Refit.RequestBuilderImplementation.<>c__DisplayClass14_0`2.<<BuildCancellableTaskFuncForMethod>b__0>d[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
--- End of stack trace from previous location ---
at Polly.Timeout.AsyncTimeoutEngine.<ImplementationAsync>d__0`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at Polly.Timeout.AsyncTimeoutEngine.<ImplementationAsync>d__0`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at Polly.AsyncPolicy.<ExecuteAsync>d__21`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at Polly.AsyncPolicy.<ExecuteAsync>d__21`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at Polly.AsyncPolicy.<ExecuteAsync>d__21`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
@zachdean do you also use Polly?
@simonrozsival , adding <UseNativeHttpHandler>false</UseNativeHttpHandler>
did not fix the problem for me :(
@haavamoa can you share a stacktrace from the app with UseNativeHttpHandler=false
?
@simonrozsival we are not using poly, but we do have a retry mechanism that is in the request pipeline. We are injecting AndroidMessageHandler
as the final request handler directly into the pipeline and the only thing special we do with it is set the automatic decompression new AndroidMessageHandler { AutomaticDecompression = DecompressionMethods.All, }
. Here is the stack trace that I was seeing. The error was surfacing in almost every http request I do in the app and appeared to be random.
System.ObjectDisposedException: ObjectDisposed_Generic ObjectDisposed_ObjectName_Name, Java.IO.InputStreamInvoker
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable)
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String, IJavaPeerable, JniArgumentValue*)
at Java.IO.InputStream.Close()
at Android.Runtime.InputStreamInvoker.Close()
at System.IO.Stream.Dispose()
at System.IO.BufferedStream.Dispose(Boolean)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.IO.Compression.DeflateStream.Dispose(Boolean)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.IO.Compression.GZipStream.Dispose(Boolean)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.Net.Http.StreamContent.Dispose(Boolean)
at System.Net.Http.HttpContent.Dispose()
at System.Net.Http.HttpResponseMessage.Dispose(Boolean)
at Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean)
at System.Net.Http.HttpResponseMessage.Dispose()
at System.Net.Http.HttpClient.HandleFailure(Exception, Boolean, HttpResponseMessage, CancellationTokenSource, CancellationToken, CancellationTokenSource)
at System.Net.Http.HttpClient.
So, we don't see any changes to AndroidMessageHandler
here:
we noticed that pinning to 34.0.95 fixed the issues.
Is it possible this actually changed the runtime version?
If you have a rollback file like:
{
"microsoft.net.sdk.android": "34.0.95/8.0.100"
}
I don't actually know what it would choose to do with the runtime. Did it downgrade the runtime to 8.0.0?
If you have a .binlog
of some of the builds, we could check.
The commit diff between 34.0.95 and 34.0.113 is: https://github.com/dotnet/android/compare/34.0.95...34.0.113
There is only one change to Mono.Android.dll
: 0315e8955cb93190832dc227aafaa90dda57a91a, which doesn't directly touch HttpClient
.
That's all I can say with any degree of confidence. :-)
I have the same question as @jonathanpeppers does.
@simonrozsival: meanwhile, within the dotnet/android change set, we also changed the dotnet/runtime that we use for unit tests in 784d320e88c29622c5fd7ee5623c18c138388740, for a runtime commit diff of: https://github.com/dotnet/runtime/compare/62304a6d7085e32672ec988835eab41d89b82e25...fd8f5b5af7d73f697fe9654ad8ed7953cd597287
which does contain HttpClient
-related changes…
I'm thus inclined to believe that something may have changed on the dotnet/runtime side.
I will see if I can provide a .binlog file.
Here's my binlog from the build where I was running the following command and setup:
> dotnet publish -bl:msbuild.binlog ConsumerApp.csproj -f net8.0-android -c Debug
> dotnet workload list git:master
Installed Workload Id Manifest Version Installation Source
---------------------------------------------------------------------
ios 17.2.8053/8.0.100 SDK 8.0.300
maui 8.0.40/8.0.100 SDK 8.0.300
android 34.0.95/8.0.100 SDK 8.0.300
@simonrozsival , this is the stack trace of the error with the following setup:
<UseNativeHttpHandler>false</UseNativeHttpHandler>
and
> dotnet workload list
Installed Workload Id Manifest Version Installation Source
---------------------------------------------------------------------
ios 17.2.8053/8.0.100 SDK 8.0.300
maui 8.0.40/8.0.100 SDK 8.0.300
android 34.0.113/8.0.100 SDK 8.0.300
Stack trace:
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.cs:line 153
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods_Invoke.cs:line 57
at Java.IO.InputStream.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/obj/Release/net8.0/android-34/mcw/Java.IO.InputStream.cs:line 116
at Android.Runtime.InputStreamInvoker.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs:line 62
at System.IO.Stream.Dispose()
at System.IO.BufferedStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.Net.Http.StreamContent.Dispose(Boolean disposing)
at System.Net.Http.HttpContent.Dispose()
at System.Net.Http.HttpResponseMessage.Dispose(Boolean disposing)
at Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean disposing) in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidHttpResponseMessage.cs:line 42
at System.Net.Http.HttpResponseMessage.Dispose()
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
@haavamoa from your .binlog
, I see the 8.0.6 runtime:
KnownRuntimePack
Microsoft.NETCore.App
TargetFramework = net8.0
LatestRuntimeFrameworkVersion = 8.0.6
...
Adding /usr/local/share/dotnet/x64/packs/Microsoft.NETCore.App.Runtime.Mono.android-arm64/8.0.6/runtimes/android-arm64/native/libmonosgen-2.0.so
In the case you've gotten it to work by installing with a rollback file, is your build using a different runtime?
UseNativeHttpHandler=false
also would indicate this is a change in the dotnet/runtime (BCL), as the code for that doesn't live in the Android workload.
UseNativeHttpHandler=false also would indicate this is a change in the dotnet/runtime (BCL), as the code for that doesn't live in the Android workload.
@haavamoa @jonathanpeppers given the last stacktrace, it is not using the managed handler, but it is still using the native one (notice Xamarin.Android.Net.AndroidHttpResponseMessage
). Are you instantiating AndroidMessageHandler
directly, @haavamoa? If so, can you change it to either SocketsHttpHandler
or HttpClientHandler
(+ UseNativeHttpHandler=false
)?
I am starting my summer vacation and I am most likely not able to give you more information until ~5 august.
but yes, we are initiating that AndroidMessageHandler @simonrozsival .
@jonathanpeppers , could you test that yourself or do you need me to do it? I can try to do so whenever I have time to it, or else it will have to wait until 5 August :p
On a sidenote, is there a binlog tool for Mac? Only one I found was for windows, and Id like to not share my binglogs for this internal project openly next time 😅
You can just dotnet build -bl
to create msbuild.binlog
. If you have CI on Azure DevOps or something, I would save these as build artifacts for every build.
Yeah, I know that, but what I wanted to know was if there was a tool for Mac to inspect the binlog files? :)
@simonrozsival I just briefly tested changing to HttpClientHandler and UseNativeHttpHandler=false. It seemed the exception is not thrown anymore.
I did not have time to do a full scale test of our production app, just a small test-app for our library that is dealing with HttpClient. But it failed when inheriting from AndroidMessageHandler, so its a good enough test IMO.
I will do a test in our production app as soon as possible. But this might be enough for you to focus on a direction of where the bug came from? :)
I tried adding binlogs to our pipeline @jonathanpeppers , but I faced some issues. I dont have time for it right now, but will look more into it once my vacation is over. Its very helpful for us in general, not only for this issue 👍🏻 I also found a mac inspection tool from the docs.
@jonathanpeppers , I keep getting this error when I am trying to get binlog files out from pipeline. I added it to dotnet publish step as this is where its relevant for the pipeline:
Error:
MSBUILD : error MSB1008: Only one project can be specified.
Full command line: '/Users/runner/hostedtoolcache/dotnet/sdk/8.0.302/MSBuild.dll -maxcpucount -verbosity:m -nologo -target:Restore --property:_IsPublishing=true --property:ArchiveOnBuild=true --property:RuntimeIdentifier=ios-arm64 --property:ApplicationDisplayVersion=1.0.0 --property:ApplicationVersion=1.0.0 -property:PublishDir=<output-path> -property:_CommandLineDefinedOutputPath=true -property:Configuration=Release -property:DOTNET_CLI_DISABLE_PUBLISH_AND_PACK_RELEASE=true <csproj-path> -bl <binlog-path>-distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/Users/runner/hostedtoolcache/dotnet/sdk/8.0.302/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/Users/runner/hostedtoolcache/dotnet/sdk/8.0.302/dotnet.dll'
Switches appended by response files:
Switch: <binlog-path>
Ive anonymized my paths btw:
dotnet publish <csproj-path> -bl <binlog-path> -f net8.0-ios -c Release -p:ArchiveOnBuild=true -p:RuntimeIdentifier=ios-arm64 -p:ApplicationDisplayVersion=1.1.0 -p:ApplicationVersion=1.1.0 -o <output-path>
@haavamoa thanks for trying the HttpClientHandler, I will try to reproduce the exception again today with all the additional information. Enjoy your vacation!
@zachdean @kmiterror would it be possible for you to run your app with this emulator or device setup that will enable extended logging?
adb logcat -G 65M
adb shell setprop debug.mono.log default,network,mono_log_level=debug,mono_log_mask=all
adb logcat -c
and after you run the app, dump the logs via:
adb logcat -d > logcat.txt
@simonrozsival I should be able to get a log dump together tomorrow
@simonrozsival I took my two store builds built with the two different android versions and did the log dump. there was not much that changed between these builds except that we used a different MAUI workload manifest (8.0.21 vs 8.0.40).
microsoft.net.sdk.android-34.0.79.txt microsoft.net.sdk.android-34.0.113.txt
@zachdean thanks for the logs. I can't see it in the logs, but I assume you didn't hit the exception in the .79 test run but you did hit it in the .113, you just didn't log the exception?
There's one interesting line in the logs:
I monodroid-net: Failed to parse cookies in the server response. System.Net.CookieException: net_cookie_parse_header
@simonrozsival I moved my local tooling to .113, but I have not been able to replicate the issue. I am not really sure how to go about figuring out what actually changed between .95 and .113. I will spend some time today trying to see if I can get v.113 to crash or throw this exception on my test device to see if I can capture the logs.
@zachdean as mentioned earlier, can you check a https://aka.ms/binlog to see the version of the runtime when it works and when it doesn't.
There are no HttpClient
-related changes in the Android workload between these versions:
@simonrozsival I have tried to replicate it on the production build locally again without success. Below are the bin logs for the build that had the error and the one immediately after that resolved the issue. One interesting thing I noticed is that the MAUI workload rollback files (8.0.21 and 8.0.60) are all pinned to android 34.0.79 instead of 34.0.95 or 34.0.113.
I am also seeing this on a newly released update in production with android tooling at 34.0.113/8.0.100. It doesn't always happen on the first network call, but for the majority of the devices that crash within the first minute it does crash on the first call, with most of remainder on the second or third call. Not able to repro crash with debug or release versions. Also see this with uptime of over an hour as well, but a lower percentage.
Rolling back to earlier android tooling 34.0.95/8.0.100 as noted by @kmiterror which will hopefully stop this until solved. dispose_crash.txt
@zachdean: thank you for binlog.zip
. It's informative. It's validated my assumption that we're using two different System.Net.Http.dll
assemblies; from android-34.0.79.binlog
:
Task "ResolveRuntimePackAssets":
…
/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.android-arm64/8.0.0/runtimes/android-arm64/lib/net8.0/System.Net.Http.dll
vs. android-34.0.113.binlog
:
Task "ResolveRuntimePackAssets":
…
/Users/runner/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.android-arm64/8.0.6/runtimes/android-arm64/lib/net8.0/System.Net.Http.dll
i.e. 8.0.0 vs. 8.0.6.
Is there some way to use the 8.0.0 packages with the 34.0.113 Android workload? I have no idea, but I imagine that would fix things.
pinging @simonrozsival …
Oddly, I don't see a tag on dotnet/runtime for 8.0.6, but I do see ones for 8.0.5 and 8.0.7, so the diffs are:
…which are quite large. However, as we only care about changes to System.Net.HttpClient.dll
, that's much smaller:
% git diff v8.0.0...v8.0.5 src/libraries/System.Net.Http | wc -l
892
% git diff v8.0.0...v8.0.7 src/libraries/System.Net.Http | wc -l
892
No changes between 8.0.5 and 8.0.7, suggesting that 8.0.6 itself may not matter.
8 commits were made:
None of these look like obvious causes to me. :-(
Meanwhile, in retrospect there is a .NET for Android-side bug here, which I'm saddened didn't occur to me until now.
Dispose()
must be idempotent; from the Implement a Dispose method docs:
To help ensure that resources are always cleaned up appropriately, a Dispose method should be idempotent, such that it's callable multiple times without throwing an exception. Furthermore, subsequent invocations of Dispose should do nothing.
This clearly is not the case here, because Stream.Dispose()
calls Close()
, which is overridden by InputStreamInvoker.Close()
, which is not idempotent (and is throwing ObjectDisposedException
): https://github.com/dotnet/android/blob/108b196d59737cac899b50739edcaf1987407bc5/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs#L59-L66
I think we need to revisit the Stream.Close()
documentation:
Notes to Inheritors
In derived classes, do not override the Close() method, instead, put all of the Stream cleanup logic in the Dispose(Boolean) method. For more information, see Implementing a Dispose Method.
i.e. we shouldn't be overriding Close()
at all?!
The way I understand this issue at the moment is:
AndroidMessageHandler.SendAsync(...)
we throw an exceptionHttpClient
catches the exception and handles it with hte HandleFailure
methodAndroidHttpResponseMessage
response objectJava.IO.InputStream
ObjectDisposedException
I'm still having hard time narrowing this down to a location in the runtime code that would somehow explain what's going on. Especially since the range of the relevant commits is so short and they don't seem related.
I wonder if the behavior of Java.IO.InputStream
is correct. Calls to Dispose
should be idempotent AFAIK and I don't think that the call to close the stream should throw if the native stream instance has been already collected.
Edit: @jonpryor I just noticed your previous comment 😄 yes, I think fixing InputStream.Close()
and InputStream.Dispose()
could fix this issue if we can't replicate the same bug with HttpClient.
While writing a PR to remove the override of Stream.Close()
, I of course wrote a unit test to try to repro this behavior, and ran across a related underlying issue: unexpected sharing of instances.
The "Dispose()
should be idempotent!" argument suggests this unit test:
var javaInputStream = new Java.IO.ByteArrayInputStream (new byte[]{0x1, 0x2, 0x3, 0x4});
var invoker = new InputStreamInvoker (javaInputStream);
invoker.Dispose ();
invoker.Dispose ();
…which works. (Yay? It is idempotent after all…?)
To get it to fail, we instead need to Dispose()
of the shared instance!:
var javaInputStream = new Java.IO.ByteArrayInputStream (new byte[]{0x1, 0x2, 0x3, 0x4});
var invoker = new InputStreamInvoker (javaInputStream);
javaInputStream.Dispose ();
invoker.Dispose ();
which results in the ~same ObjectDisposedException
call stack that was originally reported!
1) Android.RuntimeTests.InputStreamInvokerTest.Stream_Dispose_Is_Idempotent (Mono.Android.NET-Tests)
System.ObjectDisposedException : Cannot access a disposed object.
Object name: 'Java.IO.ByteArrayInputStream'.
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self)
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters)
at Java.IO.InputStream.Close()
at Android.Runtime.InputStreamInvoker.Close()
at System.IO.Stream.Dispose()
That's the good news. The bad news is that even after removing the Close()
override, my new test still fails (which isn't surprising in retrospect; expected data sharing!), and 5 other tests begin failing. Oof.
At least the failing call stack is slightly different?
6) Android.RuntimeTests.InputStreamInvokerTest.Stream_Dispose_Is_Idempotent (Mono.Android.NET-Tests)
System.ObjectDisposedException : Cannot access a disposed object.
Object name: 'Java.IO.ByteArrayInputStream'.
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self)
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters)
at Java.IO.InputStream.Close()
at Android.Runtime.InputStreamInvoker.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at Android.Runtime.InputStreamInvoker.Close()
at System.IO.Stream.Dispose()
For exposition: https://github.com/dotnet/android/blob/108b196d59737cac899b50739edcaf1987407bc5/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs#L35-L46
The problem is "now" line 40: BaseInputStream.Close()
throws ObjectDisposedException
, because BaseInputStream
was disposed "elsewhere".
Good times, still figuring things out.
All that said, now that it looks like a major part of this ObjectDisposedException
is "unanticipated data sharing", let's revisit the original stack trace:
ObjectDisposed_ObjectName_Name, Java.IO.InputStreamInvoker
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable )
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String , IJavaPeerable , JniArgumentValue* )
at Java.IO.InputStream.Close()
at Android.Runtime.InputStreamInvoker.Close()
at System.IO.Stream.Dispose()
at System.IO.BufferedStream.Dispose(Boolean )
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.Net.Http.StreamContent.Dispose(Boolean )
at System.Net.Http.HttpContent.Dispose()
at System.Net.Http.HttpResponseMessage.Dispose(Boolean )
at Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean )
at System.Net.Http.HttpResponseMessage.Dispose()
at System.Net.Http.HttpClient.HandleFailure(Exception , Boolean , HttpResponseMessage , CancellationTokenSource , CancellationToken , CancellationTokenSource )
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage , HttpCompletionOption , CancellationTokenSource , Boolean , CancellationTokenSource , CancellationToken )
Notice BufferedStream.Dispose(Boolean )
in the call stack? Where is BufferedStream
coming from? A plausible space is from AndroidMessageHandler
!
Is this responsible for data sharing? Maybe! I haven't been able to fully track dependencies yet.
I'm still musing over the above data sharing hypothesis, and I'm less convinced that it explains this issue. Consider the URLConnection.InputStream
property:
namespace Java.Net {
public abstract partial class URLConnection : Java.Lang.Object {
public virtual unsafe System.IO.Stream? InputStream {
// Metadata.xml XPath method reference: path="/api/package[@name='java.net']/class[@name='URLConnection']/method[@name='getInputStream' and count(parameter)=0]"
[Register ("getInputStream", "()Ljava/io/InputStream;", "GetGetInputStreamHandler")]
get {
const string __id = "getInputStream.()Ljava/io/InputStream;";
try {
var __rm = _members.InstanceMethods.InvokeVirtualObjectMethod (__id, this, null);
return Android.Runtime.InputStreamInvoker.FromJniHandle (__rm.Handle, JniHandleOwnership.TransferLocalRef);
} finally {
}
}
}
}
}
InputStreamInvoker.FromJniHandle()
is: https://github.com/dotnet/android/blob/3ab04df75a74470e3eec198a3edfd5df5b6ddd95/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs#L156-L169
Which, upon review with fresh eyes, implicitly causes data sharing! Untested but looks quite plausible:
var a = urlConnection.InputStream;
var b = urlConnection.InputStream;
// I don't see how this could fail, due to semantics of `InputStreamInvoker.FromJniHandle()`
Assert.AreNotSame (a, b);
// This should succeed, because the `URLConnection.getInputStream()` isn't going to create a new
// instance on every invocation…
Assert.AreSame(a.BaseInputStream, b.BaseInputStream);
a.Dispose(); // fine
b.Dispose(); // boom w/o #9103
…which kinda makes me feel bad.
That said, there's no evidence that's happening here: .InputStream
is only invoked once in AndroidMessageHandler
. Furthermore, .BaseInputStream
is never accessed, so I'm running out of possible data sharing locations…
My guess is that #9103 will "fix" the problem, in that the ObjectDispsedException
shouldn't be raised -- as BaseInputStream.Close()
is skipped entirely -- but I feel like I still don't understand how this is happening in the first place. I potentially never will.
I also started having this error sporadically after updating MAUI
dotnet workload list
Installed Workload Id Manifest Version Installation Source
---------------------------------------------------------------------------------
android 34.0.113/8.0.100 SDK 8.0.300, VS 17.10.35027.167
aspire 8.0.2/8.0.100 SDK 8.0.300, VS 17.10.35027.167
ios 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
maccatalyst 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
macos 14.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
maui 8.0.61/8.0.100 SDK 8.0.300
maui-windows 8.0.61/8.0.100 SDK 8.0.300, VS 17.10.35027.167
tvos 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
{System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Java.IO.InputStreamInvoker'.
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.cs:line 153
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods_Invoke.cs:line 57
at Java.IO.InputStream.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/obj/Release/net8.0/android-34/mcw/Java.IO.InputStream.cs:line 116
at Android.Runtime.InputStreamInvoker.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs:line 62
at System.IO.Stream.Dispose()
at System.IO.BufferedStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.IO.DelegatingStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at System.IO.StreamReader.Dispose(Boolean disposing)
at System.IO.StreamReader.Close()
at Newtonsoft.Json.JsonTextReader.Close()
at Newtonsoft.Json.JsonReader.Dispose(Boolean disposing)
at Newtonsoft.Json.JsonReader.System.IDisposable.Dispose()
at Extensions.SerializeExtensions.FromJson[IList`1](Stream Data)
at Client.Clients.<_processResponse>d__81`1[[System.Collections.Generic.IList`1[[Seller, Client, Version=1.2403.1.0, Culture=neutral, PublicKeyToken=null]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext() in D:\Mobile\ServiceManager\deps\Client\src\Clients.cs:line 370
at Client.Sellers.Find(String Company, SellerFind Terms) in D:\Mobile\ServiceManager\deps\Client\src\Sellers.cs:line 34}
base: {System.InvalidOperationException}
Message: "Cannot access a disposed object.\nObject name: 'Java.IO.InputStreamInvoker'."
ObjectName: "Java.IO.InputStreamInvoker"
Confirmed with this work loads works nice
---------------------------------------------------------------------------------
android 34.0.95/8.0.100 SDK 8.0.300, VS 17.10.35027.167
aspire 8.0.2/8.0.100 SDK 8.0.300, VS 17.10.35027.167
ios 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
maccatalyst 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
macos 14.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
maui 8.0.61/8.0.100 SDK 8.0.300
maui-windows 8.0.61/8.0.100 SDK 8.0.300, VS 17.10.35027.167
tvos 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35027.167
I am also running into this exception:
2024-08-03 23:59:02.217 +02:00 [ERR] System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Java.IO.InputStreamInvoker'.
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self)
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters)
at Java.IO.InputStream.Close()
at Android.Runtime.InputStreamInvoker.Close()
at System.IO.Stream.Dispose()
at System.IO.BufferedStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.IO.DelegatingStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at BrainDump.Core.Utilities.HttpClientExtensions.DownloadAsync(HttpClient client, String requestUri, Stream destination, IProgress`1 relativeProgress, CancellationToken cancellationToken)
at BrainDump.Core.Utilities.DownloadHelpers.DownloadFile(String source, String target, Nullable`1 timeout, IProgress`1 progress, CancellationToken cancellationToken)
at BrainDump.Core.Synchronization.OneDrive.OneDriveApi.DownloadFile(String source, String target, IProgress`1 progress, CancellationToken cancellationToken)
at BrainDump.Core.Synchronization.OneDrive.OneDriveSynchronizer.DownloadFile(Guid databaseId, String source, String target, String sourceFileName, SyncReason syncReason, Boolean reportProgress, CancellationToken cancellationToken)
at BrainDump.Core.Synchronization.OneDrive.OneDriveSynchronizer.SyncFileFromOneDriveToHost(Guid databaseId, String oneDrive, String host, SemaphoreSlim throttler, CancellationToken cancellationToken)
at BrainDump.Core.Synchronization.OneDrive.OneDriveSynchronizer.Checkout(IEnumerable`1 outOfSyncDatabaseIds, IEnumerable`1 obsoleteLocalDatabaseIds, CancellationToken cancellationToken)
at BrainDump.App.Maui.Ui.CheckoutDialog.StartSync(CancellationToken cancellationToken)
at BrainDump.App.Maui.Ui.SyncDialog.Start()
Only happens on Android. The same code runs without problems on windows.
UseNativeHttpHandler=false also would indicate this is a change in the dotnet/runtime (BCL), as the code for that doesn't live in the Android workload.
@haavamoa @jonathanpeppers given the last stacktrace, it is not using the managed handler, but it is still using the native one (notice
Xamarin.Android.Net.AndroidHttpResponseMessage
). Are you instantiatingAndroidMessageHandler
directly, @haavamoa? If so, can you change it to eitherSocketsHttpHandler
orHttpClientHandler
(+UseNativeHttpHandler=false
)?
I am back from vacation now, looks like theres effort in resolving this.
Just want to report that setting UseNativeHttpHandler=false
and making sure to not initiate AndroidMessageHandler
did the trick when running android sdk = 34.0.113
. I just tested our production app without issues when doing so.
I am also encountering this problem. Stacktrace:
[DOTNET] Error occurred: Cannot access a disposed object. [DOTNET] Object name: 'Java.IO.InputStreamInvoker'., StackTrace: at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.cs:line 153 [DOTNET] at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods_Invoke.cs:line 57 [DOTNET] at Java.IO.InputStream.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/obj/Release/net8.0/android-34/mcw/Java.IO.InputStream.cs:line 116 [DOTNET] at Android.Runtime.InputStreamInvoker.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs:line 62 [DOTNET] at System.IO.Stream.Dispose() [DOTNET] at System.IO.BufferedStream.Dispose(Boolean disposing) [DOTNET] at System.IO.Stream.Close() [DOTNET] at System.IO.Stream.Dispose() [DOTNET] at System.Net.Http.StreamContent.Dispose(Boolean disposing) [DOTNET] at System.Net.Http.HttpContent.Dispose() [DOTNET] at System.Net.Http.HttpResponseMessage.Dispose(Boolean disposing) [DOTNET] at Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean disposing) in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidHttpResponseMessage.cs:line 42 [DOTNET] at System.Net.Http.HttpResponseMessage.Dispose() [DOTNET] at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) [DOTNET] at System.Net.Http.HttpClient.
g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) [DOTNET] at My.App.ViewModels.StartPageViewModel. d__611[[System.Collections.Generic.IEnumerable1[[My.Model.MyObject, My.Model, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7baa7798e]].MoveNext() in ...\StartPageViewModel.cs:line 129
Installed workloads:
Installed Workload Id Manifest Version Installation Source
android 34.0.113/8.0.100 SDK 8.0.300, VS 17.10.35122.118 aspire 8.0.2/8.0.100 SDK 8.0.300, VS 17.10.35122.118 ios 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35122.118 maccatalyst 17.2.8078/8.0.100 SDK 8.0.300, VS 17.10.35122.118 maui-windows 8.0.61/8.0.100 SDK 8.0.300, VS 17.10.35122.118 wasi-experimental 8.0.7/8.0.100 SDK 8.0.300
In my case it doesn't occur with normal synchronous Http calls (at least that's my assessment). I have a place where I create 3 tasks that I execute with a Task.WhenAll(...), and it's when these calls get made concurrently that the crash happens.
Adding suggested
<UseNativeHttpHandler Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android'">false</UseNativeHttpHandler>
to my .csproj "fixes" the issue.
Can anyone estimate the impact of not using the native Http handler? I really don't like hacks like this.
I also face this issue...
This is for Xamarin, but it makes me a bit worried to switch to the managed handler. https://learn.microsoft.com/en-us/previous-versions/xamarin/android/app-fundamentals/http-stack
Can anyone estimate the impact of not using the native Http handler? I really don't like hacks like this.
@danielheddelin the only side-effect you might observe is a slight increase in the app bundle size of around a few hundred kB. You might also see different behavior when it comes to the default values of some headers (User-Agent
, Content-Type
). The non-native handler is the SocketsHttpHandler
which is used on all non-mobile platforms and the benefit of using it is consistency across platforms. The reason it's not the default (yet?) is mainly app size.
@Brosten this does not apply to .NET anymore. We don't use BoringSSL anymore, both the managed handler and native handler in .NET 6+ are built on top of Android/Java APIs. Is there something in particular that still makes you worried to switch to the managed handler?
@simonrozsival: Thanks for your reply! Now I'm facing the options to either go singelton or using a managed http client. (Or both...) What way would you recommend?
Also running into this after migrating from XA to .NET for Android, will see if I can set up a repro later today or over the weekend.
Stack trace:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Java.IO.InputStreamInvoker'.
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.cs:line 153
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualInt32Method(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods_Invoke.cs:line 502
at Java.IO.InputStream.Read(Byte[] b, Int32 off, Int32 len) in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/obj/Release/net8.0/android-34/mcw/Java.IO.InputStream.cs:line 247
at Android.Runtime.InputStreamInvoker.Read(Byte[] buffer, Int32 offset, Int32 count) in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs:line 89
at System.IO.Stream.<>c.<BeginReadInternal>b__38_0(Object <p0>)
at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__281_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at System.IO.BufferedStream.CopyToAsyncCore(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
at System.Net.Http.HttpContent.<WaitAndReturnAsync>d__82`2[[System.Net.Http.HttpContent, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.String, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at xxx.Api.WebServiceDelegationHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /xxx.Api/Services/WebServiceDelegationHandler.cs:line 128
Android framework version
net8.0-android
Affected platform version
Android SDK 34.0.113
Description
Hello, and thank you for this amazing project.
We have a mobile application running MAUI in hospitals in Norway, where Android is one of the most used platforms. We needed to deliver a new version this week. Last Wednesday / Thursday we noticed that we started getting sporadic http client disposal messages. After some time we figured out that the only changes to the app was the Android SDK version delivered by .NET . 34.0.113 was released, and we noticed that pinning to 34.0.95 fixed the issues.
After a lot of times, I've actually not been able to give you a reproducible project , and due to internal policies in my company I am unable to share my project.
But I can give you a brief explanation of the architecture surrounding http client in the project:
The times I often see it is when I navigate between two pages that is using the same http client, but is using it against different request urls.
Here is the exception we keep seeing:
I understand that this a long shot as I do not have a reproducible project, but I hope that anyone in the team / community can reach out and see if they might have broken something, or have an idea on what we are potentially doing wrong.
Appreciate your time!
Steps to Reproduce
None unfortunately.
Did you find any workaround?
For now, we pin the SDK version by using the rollback feature of
dotnet workload install
, which saved our delivery, but we plan to upgrade every bits and pieces soon, so this will soon be an issue for us again.Relevant log output
No response