Closed StefAnglr closed 3 years ago
Thank you for your feedback!
For us to investigate this further, could you please provide your full build logs, crash reports (if any), and small sample project. The sample is the best way for us to investigate and reproduce the issue.
To get full build logs just set the log verbosity to diagnostic at the following locations:
We look forward to hearing from you!
It's not very reproducible, so I'm not sure if a sample project would do any good - I've only seen it in person once, which allowed me to narrow down the null reference. I can contrive a simple app that uses an HttpClient with a native handler to hit an API if it's necessary in order for the issue to be investigated.
I've attached full build logs from a local build along with crash reports from our production app. I've included one report from another NSUrlSessionHandler NRE crash, this one in DidCompleteWithError
, which also calls a method on inflight.CancellationTokenSource
without null checking it first, so it could be the same root cause.
@StefAnglr Thanks for investigating and attaching the crash logs.
A sample project replicating the conditions of the crash would be useful. Even if we can't reproduce, it'll at least give a better idea of what conditions make the crash possible.
For visibility:
Xamarin Exception Stack:
System.NullReferenceException: Object reference not set to an instance of an object.
at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) <0x102e34eb0 + 0x0011c> in <75e8f2cafde249969958834208668188#bc0f3fa4a04040f4e9892c3a8822ddde>:0
We have not received the requested information.
If you are still experiencing this issue please provide all the requested information then click the Reopen Issue button.
Thanks!
Sorry for the delay on this. Attached is a simple one-screen iOS app that uses HttpClient in a similar manner to our application. We implement timeouts using our own CancellationTokenSource due to the native HttpClient implementations throwing different platform-specific exceptions on timeouts (mainly an issue on Android).
In the meantime, we have copied the NSUrlSessionHandler code into our own application and added the exception handling ourselves to prevent the crashes from happening again. Once that build is released to the public we may have more useful logging from those handled exceptions.
I don't see a "Reopen Issue" button - I'm guessing I don't have the permissions for that.
@StefAnglr strange, you should have the rights to reopen. Anyway, I'll do it.
Regarding the issue, it looks very interesting, I have a few questions about the repo application:
I will get to this ASAP, do please take into consideration that we are working on the Xcode 12 changes and I might take a little longer than I would like.
PS: @whitneyschmidt I'll take over this.
Thanks for the reply. I understand if this is low priority due to the low reproduction rate, and I think our workaround should prevent further crashes once released, but I appreciate whatever attention you can give it.
In our application, we use our CancellationTokenSource approach to cancel any UI-blocking requests after 15 seconds, and any background requests after 60 seconds, so I don't think we ever allow the HttpClient to cancel the request itself. Our user reports suggest it is a background request that is causing the crash.
The sample doesn't reproduce the issue, nor can we reliably reproduce the issue in our own app; the sample just provides an example of our approach to using the HttpClient. I have only seen it happen once while debugging. Normally I wouldn't open an issue for something I couldn't reproduce, but the debugger showed the NullReferenceException being thrown within NSUrlSessionHandlerDelegate.SetResponse on the CancellationTokenSource, and the crash reports we've received from production have shown the same, along with an ObjectDisposedException on the same line, so I thought the fix might be evident without the reproduction.
This is happening in the Stable channel. If it were easier to reproduce I'd try with the Preview channel and see if it was still happening there.
Here's a few more crash logs from our latest build. I tried to use our own copy of NSUrlSessionHandler with these exceptions caught, but I've either misconfigured something, or a dependency is still using the Xamarin version. NSUrlSessionHandlerCrashLogs.zip
In Microsoft AppCenter, we also see this NullReferenceException crash in our app. The crash points into NSUrlSessionHandler when accessing the CancellationTokenSource of this "inflight" object. We have installed Xamarin.iOS 13.20.2.2 in Visual Studio for Mac. The app is a Xamarin Forms App and we use the System.Net.Http.HttpClient class inside our shared code.
Exception Type: SIGABRT
Exception Codes: #0 at 0x183288df0
Crashed Thread: 12
Application Specific Information:
*** Terminating app due to uncaught exception 'SIGABRT', reason: 'Object reference not set to an instance of an object'
Xamarin Exception Stack:
System.NullReferenceException: Object reference not set to an instance of an object
at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) [0x00023] in /Library/Frameworks/Xamarin.iOS.framework/Versions/13.20.2.2/src/Xamarin.iOS/Foundation/NSUrlSessionHandler.cs:663
This is the source code at the crash location in /Library/Frameworks/Xamarin.iOS.framework/Versions/13.20.2.2/src/Xamarin.iOS/Foundation/NSUrlSessionHandler.cs (line 663):
:
655 // this can happen if the HTTP request times out and it is removed as part of the cancellation process
656 if (inflight != null) {
657 // set the stream as finished
658 inflight.Stream.TrySetReceivedAllData ();
659
660 // send the error or send the response back
661 if (error != null || serverError != null) {
662 // got an error, cancel the stream operatios before we do anything
663 inflight.CancellationTokenSource.Cancel (); // <-- crash because CancellationTokenSource is null
664 inflight.Errored = true;
:
Why there is no "null" check here and in all other places where the CancellationTokenSource is accessed? Is this a bug in Xamarin.iOS? The app should really not crash when there is a network issue.
In our Xamarin Forms shared code in an utils class we simply create a HttpClient instance and send a request:
HttpClient myHttpClient = new HttpClient();
HttpResponseMessage response = await myHttpClient.SendAsync(request, CancellationToken.None);
string responseString = await response.Content.ReadAsStringAsync();
I also tried to put a try/catch around the HttpClient.SendAsync() but this kind of crash will not be catched? Don't know why.
How can I avoid this crash? Can I do something in my application code or have I wait till the issue is fixed in a new release?
I believe it's not possible to catch the exception in your own code because the exception is thrown in a networking callback. We've tried to work around the issue by using our own customized version of NSUrlSessionHandler, but we have dependencies that don't easily support customizing their HttpClients.
This NullReferenceException in DidCompleteWithError() which I reported above is more and more a problem for us. The number of crashes is increasing in AppCenter. For info, we upgraded from Xamarin.iOS 13.6.0.12 to 13.20.2.2. I need a workaround as soon as possible.
@StefAnglr: You said you wrote a customized NSUrlSessionHandler? How did you do this? I use the HttpClient in my shared code, so I cannot initialize it with a custom Handler, because NSUrlSessionHandler is platform specific. Is it possible to instantiate a HttpClient via the DependencyService for Android and iOS?
I also think about to override the DidCompleteWithError() callback and add this null-check by myself, but I don't know how.
Would be glad if someone has a workaround for this.
I made some analyzing and want to share the infos here. I duplicated the file "NSUrlSessionHandler.cs" and renamed the class to "MyNSUrlSessionHandler" and call the HttpClient constructor with this copied handler. This way it is possible to debug the code. I wanted to find out why the CancellationTokenSource is "null" at some places which then leads to the NRE in SetResponse() or DidCompleteWithError(). In our app we use the Azure Application Proxy (AAP) to access the backend server (IIS) in the company intranet. I forced a Gateway Timeout (HTTP Status code 504) in the AAP by setting a breakpoint in Visual Studio debugging the backend IIS. That way the response from IIS will timeout and after 90 seconds the AAP sends a GatewayTimeout (504) back to my Xamarin iOS app. In the Xamarin app now the DidReceiveResponse callback will be called and the GetInflightData() will be called:
Inside the GetInflightData() method the inflight is fetched by using the task object. We see that the "IsCancellationRequested" flag is "True" and the CancellationTokenSource is still valid. Because the flag is true a Cancel() will be called on the task instance.
When I step further in the debugger, you see that the CancellationTokenSource is now "null". I assume because of the Cancel() the CancellationTokenSource was disposed and set to null):
After the GetInflightData() returns the inflight in DidReceiveResponse() there is an access to inflight.CancellationTokenSource in line 644 in the next screenshot, which leads to a NullReferenceException (line 687):
Because the DidReceiveResponse() method has a try/catch this NRE is handled and the app doesn't crash. But what is when the DidCompleteWithError() or the DidReceiveData() is called directly without DidReceiveResponse()? In DidCompleteWithError() and DidReceiveData() the method GetInflightData() is called to get the inflight, the Cancel() will be called inside GetInflightData() which leads to a "null" in CancellationTokenSource. These 2 methods have no try/catch and so the SetResponse() call inside DidReceiveData() or the line "inflight.CancellationTokenSource.Cancel();" inside DidCompleteWithError() will throw a NullReferenceException and the app crashes.
@mandel-macaque Can you please have a closer look into this issue? Perhaps my infos above help you. The question is why your code expect that the CancellationTokenSource is never "null" when we see here that it will be set to "null" whenever the task will be cancelled?
Been hunting this for a while and found this thread. We have around 50 crashes per day because of this. A solution would be most welcome
@SuperMario77 Thanks for investigating this and sorry I missed your question earlier.
To try to work around the crash, you can add the exception handling to your MyNsUrlSessionHandler
class, and then register that class with your DependencyService as the implementation for IHttpClientProvider
on iOS (and register the standard Xamarin.Android.Net.AndroidClientHandler
on Android).
Then in your shared code, you can resolve IHttpClientProvider
and pass it to the HttpClient
constructor. How you do this will depend on how you instantiate HttpClient
.
@mandel-macaque @rolfbjarne Is there any update for this issue? This NRE is more and more a problem for us and it seems that other developers have the same problem! We have cloned the NSUrlSessionHandler.cs and fixed the locations where CancellationTokenSource is not checked for "null" and use now only a HttpClient instance with our customized NSUrlSessionHandler. This helped a little bit.
BUT there are still 3rd party libraries in our app like MSAL library (Microsoft Identity Client) for doing Azure AD authentication which still use the HttpClient with native NSUrlSessionHandler. And the NREs in NSUrlSessionHandler still happen a lot, which means a crash of the app, which means unsatisfied customers.
Please take again a look into this issue.
@mandel-macaque Here is an iOS crash dump of the NRE in NSUrlSessionHandler. At the bottom of the file you will see the Thread 22 where the Microsoft Identity Client Library uses the native HttpClient (without our cloned "fixed" NSUrlSessionHandler) which leads to the NRE and to an app crash. So even in an official Microsoft library (MSAL) this NRE occurs, so IMHO this issue should be solved asap, because the MSAL library is surely used in a lot of Xamarin iOS apps to do Azure AD authentication. In general, this exception can happen in all 3rd libraries which uses HttpClient with native NSUrlSessionHandler implementation.
Sorry for the delay. I had to work on other issues but I have bump this to the top of my queue. Will get a fix ASAP.
@SuperMario77 @StefAnglr @ThomasHoest I have created a fix for the issue. If you check the PR you will see an explanation of why we had a crash.
tldr; The cancellation of the task in the inflight data had a ripple effect that disposed the instance of the infligh data. If the task is cancelled the result of that method should be null to ignore the rest of the delegate methods.
Looks like a fun bug to track down. Thanks for investigating!
@StefAnglr @ThomasHoest @SuperMario77 You can find the following packages with the fix:
Can you please test and confirm is fixed and I'll download the fix and will backport to the current stable branch.
I never had much luck reproducing the issue on demand. Knowing what you do now about the bug, what would you suggest? Send a network request that will time out, and cancel the request immediately?
yes, but is going to be hard, since.. well, the bug is fixed, so you should not be able to repo. @SuperMario77 can you give it a go?
@ThomasHoest maybe you can confirm the fix?
@mandel-macaque Thank you so much for finding a fix for this issue. Unfortunately, at the moment I have not the possibility to confirm your fix and because this issue happens very randomly on the users devices it is even more difficult to test your fix.
Landing this on main for it to be picked up in a future release.
@mandel-macaque i looked at the latest release and i don't see this item on it. Do you know when it will be on there?
@ahhassan3 I need to check with the release manager, it was landed in main but was not bacported. @dalexsoto do you have any ideas?
@mandel-macaque i just looked at the source files for the latest xamarin ios release (xamarin-ios-14.14.2.5) and the fix wasn't on there. i don't understand why this isn't getting released
Hi @mandel-macaque, any update on how to resolve this crash issue please? I'm using Xamarin.iOS 14.8.0.3 installed in visual studio & Xamarin.iOS 14.4.1.3 in my paired Mac
any fix available?
This is a long-standing issue for us too. Multiple crashes per day among our userbase, which of course are interpreted as instability in our software (I suppose they are correct). Since a fix has obviously been engineered, why is it not in our hands?
Think I've jsut had this issue occur myself, but would be good to know if thats the case.
Had the following Exception occur, I think when I cancelled out a REST request.
SIGABRT: Object reference not set to an instance of an object
NSUrlSessionHandler+NSUrlSessionHandlerDelegate.SetResponse (System.Net.Http.NSUrlSessionHandler+InflightData inflight)
/Users/builder/azdo/_work/1/s/xamarin-macios/src/Foundation/NSUrlSessionHandler.cs, line 684
This the same issue? I'm on the latest Stable Visual Studio branch
Apple Developer Tools
Xcode 12.4 (17801)
Build 12D4e
Xamarin.Mac
Version: 7.8.2.5 (Visual Studio Enterprise)
Hash: 3836759d4
Branch: d16-9
Build date: 2021-02-10 17:56:43-0500
Xamarin.iOS
Version: 14.14.2.5 (Visual Studio Enterprise)
Hash: 3836759d4
Branch: d16-9
Build date: 2021-02-10 17:56:44-0500
Anyone able to confirm if this is the same area/line/crash as being reported, hard to tell if the above is fixed in an actual release as people are reporting it isn't but the ticket is closed several months ago.
Cheers for any advice 👍
Which is this closed and obviously not fixed?
2.500 crashes on this every month. When will the fix be released?
This is so not closed/released...
Any update on when this fix will be released?
Having issues here too - a number of crashes every day - would be great to have a fix ASAP
+1
Having exactly the same issue, lots of crashes daily because of this error - this needs to be fixed.
Also seeing many NRE that are very likely to be caused by this issue. I don't understand why this issue has been closed even though it's not been released for quite a while now...
This issue is closed because the issue has been fixed and released. The only person that has provided a Xamarin.iOS version has been @IainS1986 with version 14.14.2.5, hash 3836759d4 which as you can see in the source code https://github.com/xamarin/xamarin-macios/blob/3836759d4/src/Foundation/NSUrlSessionHandler.cs#L543 does not have the fix.
The d16-9 branch is not longer the stable branch, since we moved to the d16-10 branch. @mduchev @alexanderdibenedetto @tipa the latests release of Xamarin.iOS is https://download.visualstudio.microsoft.com/download/pr/414d1675-ca0b-4b5b-8a4e-26b693883581/5e50228d1498caf85151b6d6cfec6fd9/xamarin.ios-14.20.0.1.pkg
Please test the stable version, if the issue reproduces, do not simple add a +1, that is not giving me enough information to do any work, even though you want to help it does the opposite :(
If the above pkgs does not have the issue resolved, please provided the full environment information and the symbolicated exception.
@adams-family as I have stated in your issue, the version you are using is 14.8.0.3 (c51fabee8) which is much older than the fix (10th of December when the fix landed on the 22 second of March).
@IainS1986 Please update to the provided pkg, I missed your question because closed issues are muted by default, please next time feel free to @ me directly in the issue so that I can responde faster, I take the blame for not looking at conversations in closed issues.
(Edit: updated the pkg to d16-10 which is the new stable)
@mandel-macaque Thanks for your quick response, I've posted a comment in the issue I've created.
I'm having the same issue, Xamarin.iOS version 15.2. Only affects 1% of users per week - probably one in a hundred requests. I'll see if I can make a sample project after the new year
Also still seeing the issues in 15.x. It is not fixed. 1% seems about right looking at our crash numbers
Had this happen to me today Stack trace from Visual Studio, running on iOS Simulator 15.2
Unhandled managed exception: Object reference not set to an instance of an object (System.NullReferenceException)
at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.SetResponse (System.Net.Http.NSUrlSessionHandler+InflightData inflight) [0x00032] in /Users/builder/azdo/_work/1/s/xamarin-macios/src/Foundation/NSUrlSessionHandler.cs:781
at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidReceiveData (Foundation.NSUrlSession session, Foundation.NSUrlSessionDataTask dataTask, Foundation.NSData data) [0x00018] in /Users/builder/azdo/_work/1/s/xamarin-macios/src/Foundation/NSUrlSessionHandler.cs:743
Still our highest cause for crashes on iOS. About 250 per day. This should be re-opened.
Still here. @mandel-macaque what do you need to look into this again?
Main Global terminating exception: Object reference not set to an instance of an object stack at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) <0x101ffd2c0 + 0x00074> in <a7229a51752a4b23a2321c4878d035f4#b2bc9717d735d13fb9be7e6434b655da>:0 System.NullReferenceException Object reference not set to an instance of an object at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) <0x101ffd2c0 + 0x00074> in <a7229a51752a4b23a2321c4878d035f4#b2bc9717d735d13fb9be7e6434b655da>:0 | System.NullReferenceException Object reference not set to an instance of an object at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) <0x101ffd2c0 + 0x00074> in <a7229a51752a4b23a2321c4878d035f4#b2bc9717d735d13fb9be7e6434b655da>:0
We are getting these too, out in the wild, a few per day. Sometimes an NRE, sometimes the below:
System.ObjectDisposedException: The CancellationTokenSource has been disposed. at System.Threading.CancellationTokenSource.ThrowObjectDisposedException () <0x10089bfc8 + 0x0003c> in <25bf495f7d6b4944aa395b3ab5293479#9a38be3e91be429a650558472b7ee5fb>:0 at System.Threading.CancellationTokenSource.ThrowIfDisposed () <0x10089bf98 + 0x0002f> in <25bf495f7d6b4944aa395b3ab5293479#9a38be3e91be429a650558472b7ee5fb>:0 at System.Threading.CancellationTokenSource.Cancel () <0x10089baf4 + 0x00013> in <25bf495f7d6b4944aa395b3ab5293479#9a38be3e91be429a650558472b7ee5fb>:0 at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession , Foundation.NSUrlSessionTask , Foundation.NSError ) <0x100ee0564 + 0x00077> in <5b08e01cb6df409eb2fea153e6177e5d#9a38be3e91be429a650558472b7ee5fb>:0
Same here! We are also many crashes reported in MSAppcenter: NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error)
I got these 2 lines while i was investigating a breakpoint - i believe the call timeout:
API MISUSE: NSURLSession delegate System_Net_Http_NSUrlSessionHandler_NSUrlSessionHandlerDelegate: <System_Net_Http_NSUrlSessionHandler_NSUrlSessionHandlerDelegate:
and
API MISUSE: dataTask:didReceiveResponse:completionHandler: completion handler not called
Not sure if these 2 lines help
Steps to Reproduce
Unknown. It happened to me once while debugging, and we have several crash reports in App Center that look to be the same issue, but I haven't seen a consistent reproduction. This is in an iOS app running on iOS 13.
Expected Behavior
NSUrlSessionHandlerDelegate.SetResponse
doesn't crash the app with a NullReferenceExceptionActual Behavior
NSUrlSessionHandlerDelegate.SetResponse
crashes the app with a NullReferenceException.inflight.CancellationTokenSource
was null on this line:Environment