Closed SOFSPEEL closed 4 years ago
@mandel-macaque - Is this the known "fixed" issue or another can of worms?
This is not a designer bug - I fat fingered the category trying to hit ios and going too fast.
@chamons looks like a diff issue, but I'd be surprise this will happen. Do we have a code sample to reproduce it. The cancelation, the way I understand it is thread safe. I could add an extra check, but it would be nice to have a sample before adding more locks.
Also, we have https://github.com/xamarin/xamarin-macios/pull/6537 waiting to be landed.
I also wanted to add that this is not an isolated incident, this has occurred with 20 different users over a two week period:
We're experiencing this issue in our production app as well
Having this issue too in our Production app since our last update in Apple Store 17 days ago
There's not enough information here to determine if this is a threading issue or if the inflightRequests collection is being modified by the same thread when calling r.CompletionSource.TrySetCanceled
.
I think this is a better attempt at a fix:
void BackgroundNotificationCb (NSNotification obj)
{
foreach (var r in inflightRequests.Values.ToArray ()) {
r.CompletionSource.TrySetCanceled ();
}
}
@mandel-macaque since we don't have a test case, could you create a package with this fix so that it can be tested?
Hi! I have the exactly the same Crash report from AppCenter but I'm getting it when app not in memory (killed) and device locked .... and we have voip pushes that wakeup app... and it crashes in 99% ... so could be reproducible very well in my case
device has fingerprint lock .... when I open it after ReportInComingCall CallKit and I could see app few seconds ... then I have that crash and app restarts ... I could take video
PS: when device has no pass code or no fingerprint ... it crashes in 50%. and only when app not in memory (i.e. killed)
also we having this crash a lot ... maybe related to this one ...
@mandel-macaque what could we do?
One of our production apps is having the same issue. Unfortunately, we have not been able to reproduce this error.
Though @ideveloperaz seems to be on to something here.
Sorry, I was moving countries and could not reply fast enough. I will post a package (or a link to it) with the collection being locked to be tested. We already have the PR ready.
My statistics concerning the issue. Let's force the fix please.
I have provided a fix that will ensure that this does not longer happen. The proposed fix is in a PR that will generate a package, so that you can use it for testing and confirming the fix. There was not need for an extra lock, but for a temp collection to transverse. If approved, I will make sure that the fix is back ported two the next coming release.
Again, sorry for the delay, was out of work, is not a excuse but I want to give context for the delay :)
Packages can be found here:
Hello Everyone!
Since I'm the only one who could reproduce that crash here ... hereby I honestly confirm that that issue is fixed :-)
Good job @mandel-macaque !
But I'm having and was having different issue here with NSNotificationCenter.RemoveObserver I believe here is similar issue
@ideveloperaz I already have a PR in the flight for that too :) https://github.com/xamarin/xamarin-macios/pull/6537
Great @mandel-macaque :-) where we could find packages with all that issues altogether ? 😀 Today is really happy Friday ))
@ideveloperaz pkg is building, I made sure that you will get both issues fixed and I'll post them here for you to confirm :)
@ideveloperaz can you please take a look at the following .pkg and let me know if they are good for you:
Oh thank you a lot, Man! But I could do that during weekend, that issue with NSNotificationCenter not could be reproduced like prev one.On 30 Aug 2019 6:57 pm, Manuel de la Pena notifications@github.com wrote:@ideveloperaz can you please take a look at the following .pkg and let me know if they are good for you: xamarin.iosxamarin.mac
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.
@ideveloperaz no worries, weekends are weekend, and you help is really welcomed :). I'll probably land this in master, but if you can do a +1 would be great.
Thank you a lot :-) I will do my best. But based on my monitoring of that issue, this could take some time to completely ensure that this issue fixed.On 30 Aug 2019 7:20 pm, Manuel de la Pena notifications@github.com wrote:@ideveloperaz no worries, weekends are weekend, and you help is really welcomed :). I'll probably land this in master, but if you can do a +1 would be great.
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.
@ideveloperaz can you please take a look at the following .pkg and let me know if they are good for you:
When will the package with the fix available in the visual studio update? Are your linked package safe to install if we currently use 12.14.x?
@ideveloperaz can you please take a look at the following .pkg and let me know if they are good for you:
When will the package with the fix available in the visual studio update? Are your linked package safe to install if we currently use 12.14.x?
@StingrayFE, I could test .315 and it worked for me, regarding .317 I could never reproduce it myself only saw in crash logs so still keeping eyes on it, but could not see that issue as far
@ideveloperaz can you please take a look at the following .pkg and let me know if they are good for you:
When will the package with the fix available in the visual studio update? Are your linked package safe to install if we currently use 12.14.x?
I could not answer your question regarding when fixes will be available in VS because I dont work over fixes. But you could install that packages over 12.14.* ... was safe to install for me
Hi @mandel-macaque please kindly merge all fixes in that fix with #6762 #6916
why this fix (#6764) was not in current VS update?
may I have package with all that fixes in one place? or any ideas when #6764 will be in VS stable update?
This issue is still occurring for us. Let me be clear: our app is crashing, and our app is nowhere in the stack trace, because of this bug in Microsoft's code.
What is the status of the fix please?
@divil5000 which versions are you currently using? Can you get the version information from your IDE?
=== Visual Studio Professional 2019 for Mac ===
Version 8.3.11 (build 1) Installation UUID: fd515159-22b2-4783-b0b7-011a4038bba0 GTK+ 2.24.23 (Raleigh theme) Xamarin.Mac 5.16.1.24 (d16-3 / 08809f5b)
Package version: 604000208
=== Mono Framework MDK ===
Runtime: Mono 6.4.0.208 (2019-06/07c23f2ca43) (64-bit) Package version: 604000208
=== NuGet ===
Version: 5.3.0.6192
=== .NET Core SDK ===
SDK: /usr/local/share/dotnet/sdk/3.0.101/Sdks SDK Versions: 3.0.101 3.0.100 2.1.701 2.1.505 2.1.504 2.1.302 2.1.301 2.1.4 2.0.0 MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/6.4.0/lib/mono/msbuild/Current/bin/Sdks
=== .NET Core Runtime ===
Runtime: /usr/local/share/dotnet/dotnet Runtime Versions: 3.0.1 3.0.0 2.1.14 2.1.13 2.1.12 2.1.9 2.1.8 2.1.2 2.1.1 2.0.5 2.0.0
=== Xamarin.Profiler ===
Version: 1.6.12.26 Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler
=== Updater ===
Version: 11
=== Xamarin.Android ===
Not Installed
=== Microsoft Mobile OpenJDK ===
Java SDK: Not Found
Android Designer EPL code available here: https://github.com/xamarin/AndroidDesigner.EPL
=== Android SDK Manager ===
Version: 1.4.0.65 Hash: c33b107 Branch: remotes/origin/d16-3 Build date: 2019-11-19 20:33:22 UTC
=== Android Device Manager ===
Version: 1.2.0.116 Hash: d2b2af0 Branch: remotes/origin/d16-3 Build date: 2019-11-19 20:33:42 UTC
=== Apple Developer Tools ===
Xcode 11.2.1 (15526.1) Build 11B53
=== Xamarin.Mac ===
Version: 6.6.0.12 (Visual Studio Professional) Hash: e3c2b406d Branch: xcode11.2 Build date: 2019-11-01 00:12:07-0400
=== Xamarin.iOS ===
Version: 13.6.0.12 (Visual Studio Professional) Hash: e3c2b406d Branch: xcode11.2 Build date: 2019-11-01 00:12:08-0400
=== Xamarin Designer ===
Version: 16.3.0.256 Hash: 8a223bfd7 Branch: remotes/origin/d16-3 Build date: 2019-11-01 21:02:02 UTC
=== Xamarin Inspector ===
Version: 1.4.3 Hash: db27525 Branch: 1.4-release Build date: Mon, 09 Jul 2018 21:20:18 GMT Client compatibility: 1
=== Build Information ===
Release ID: 803110001 Git revision: 6ee6ad2ec46ae5a08a1999ee4c815ac656a35b91 Build date: 2019-12-05 16:09:27+00 Build branch: release-8.3 Xamarin extensions: 56bd70ef2e327f71c615cfc29a47fd50685fadcb
=== Operating System ===
Mac OS X 10.14.4 Darwin 18.5.0 Darwin Kernel Version 18.5.0 Mon Mar 11 20:40:32 PDT 2019 root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
This was landed in xcode11 via
https://github.com/xamarin/xamarin-macios/pull/6880
and 13.6.0.12 is the current stable.
Could you please provide a stack trace at minimum and preferably a sample. This is the first report of regression, and I want to confirm that it is the same issue.
Here's our telemetry. I'm sure you're well aware that this isn't the sort of bug that can be easily reproduced on dev machines; we are seeing it hit our users in the field maybe 2 times per day, with tens of thousands of installations.
Version: 3.13.1.26610 MonoTouch: 13.6.0 iOS: 13.3 Hardware: iPhone12,1 System.InvalidOperationException: Collection was modified; enumeration operation may not execute. at System.Collections.Generic.Dictionary`2+ValueCollection+Enumerator[TKey,TValue].MoveNext () <0x1031c55d8 + 0x000c3> in <0e400f473a3e4716a829e46ba2f1f0d6#75df90ebb0348df8ab8974ad0820e7de>:0 at System.Net.Http.NSUrlSessionHandler.BackgroundNotificationCb (Foundation.NSNotification ) <0x1032c3878 + 0x0008b> in <e139e7c752b84c85bc1cc3eeae1725fa#75df90ebb0348df8ab8974ad0820e7de>:0 at Foundation.InternalNSNotificationHandler.Post (Foundation.NSNotification ) <0x103305e64 + 0x00017> in <e139e7c752b84c85bc1cc3eeae1725fa#75df90ebb0348df8ab8974ad0820e7de>:0 at (wrapper managed-to-native) UIKit.UIApplication.UIApplicationMain(int,string[],intptr,intptr) at UIKit.UIApplication.Main (System.String[] , System.IntPtr , System.IntPtr ) <0x1032e80b0 + 0x0003f> in <e139e7c752b84c85bc1cc3eeae1725fa#75df90ebb0348df8ab8974ad0820e7de>:0 at UIKit.UIApplication.Main (System.String[] , System.String , System.String ) <0x1032e8008 + 0x00053> in <e139e7c752b84c85bc1cc3eeae1725fa#75df90ebb0348df8ab8974ad0820e7de>:0 at Divelements.SkyDemon.Application.Main (System.String[] args) <0x102b0232c + 0x000bb> in <8bde7be67e974f7097924a2dcaf1f8a3#75df90ebb0348df8ab8974ad0820e7de>:0
@divil5000 until we find the root cause, please do the following in you application:
NSUrlSessionHandler handler = new NSUrlSessionHandler() { BypassBackgroundSessionCheck = true}
That will stop the crash.
Regarding your crash, is that the full stack trace> Can you provide a small sample of your code?
So we did some digging, and it turns out we did multiple passes at fixing this issue, one which made it rarer and one that fixed it completely. Due to release timing, the first fix is in stable today, and another is in d16-5, and upcoming release early next year.
As @mandel-macaque noted, you can work around this until that fix goes out.
Thanks for the report and feel free to reach out if you have any questions.
That is indeed a full stack trace. Thanks for the example line of code; can you explain what it does and its potential for side effects please? We are never explicitly instantiating NSUrlSessionHandler (we just use HttpClient) so it isn't clear where we should be using that suggested workaround.
This is presumably a related issue? Again, we see this thrown, crashing our app when our app isn't even in the stack trace, on a near-daily basis.
ObjectDisposedException at System.Threading.CancellationTokenSource.ThrowObjectDisposedException () <0x100fcfbcc + 0x00038> in <0e400f473a3e4716a829e46ba2f1f0d6#75df90ebb0348df8ab8974ad0820e7de>:0 at System.Threading.CancellationTokenSource.Cancel () <0x100fcf6e8 + 0x0001f> in <0e400f473a3e4716a829e46ba2f1f0d6#75df90ebb0348df8ab8974ad0820e7de>:0 at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession , Foundation.NSUrlSessionTask , Foundation.NSError ) <0x1011a4474 + 0x0004f> in :0
That is the full stack trace.
@divil5000 The one-liner will remove the background notification that will cancel the request when you application goes to the background and the NSUrlSession was not created with a background configuration. This was added as a workaround to an issue that we found in the mono runtime threadpool in certain architectures in which the Treadpool will not recover correctly and the async callbacks would not work correctly when the application gets to the front.
You will only hit the issue of the tread pool in a small number of old devices, so with the one liner you will remove the crashes in most if not all modern devices and you might see issues in older ones. If I record correctly, 5S was the issue.
Thank you. It still isn't clear where that one-lines goes, though; I am not instantiating that class anywhere. Do I have to integrate it with my HttpClient calls somehow?
We're experiencing exactly the same problems as @divil5000 does:
@divil5000, answering your question about where to put this one-liner, you should do it when instantiating HttpClient:
_httpClient = new HttpClient(new NSUrlSessionHandler() { BypassBackgroundSessionCheck = true });
If you're creating your HttpClient in Core, you might want to create platform-specific HttpClientFactories or something like that
By the way, I also have some more questions about this new BypassBackgroundSessionCheck flag. In our app, we have the problem with Http RefreshToken call. When the call is started and the app is backgrounded right away, the payload (that actually contains the token) is not attached to the Http response or is attached only partially. With BypassBackgroundSessionCheck flag set, it looks like the problem is gone, the content is always attached to the HTTP call. So, there are several different questions I want to ask:
Thank you.
Unfortunately my Visual Studio for Mac reports being fully up-to-date and yet BypassBackgroundSessionCheck is NOT a member of NSUrlSessionHandler.
What on earth is going on? You've had this bug in your code for months with people running into it and yet it doesn't seem to be being fixed. Please note that we cannot use beta/non-stable software.
Sometimes I wonder if we are the only company using Xamarin for iOS in an app with tens of thousands of deployments being used every day by the public.
@divil5000, you're not the only one, but it looks like there are not a lot of apps using Xamarin right now, that's true. I've just updated Xamarin iOS to the latest Stable version and the flag is not there for me as well. Looks like it's still only available in Beta right now
I'm not sure if that's connected, but looks like it might be:
Unfortunately, I can't provide you the version of tools that we were using when creating this build, but I definitely know that it was right before updating to xCode 11, so that might be already fixed
Full crash log: crashlog.txt
I can see that BypassBackgroundSessionCheck is included in the latest Xamarin.iOS release, thanks for that. But some questions are still open, can you please address them:
Crash report from AppCenter is:
Note that I suspect the crash occurs because the following code in NsUrlSessionHandler is not locked:
https://github.com/xamarin/xamarin-macios/blob/master/src/Foundation/NSUrlSessionHandler.cs
So this:
Should be changed to (I think):
Steps to Reproduce
Expected Behavior
Actual Behavior
Environment
Build Logs
Example Project (If Possible)