False OOM crash reports

[prabhuamol]

Describe the bug

We are seeing a lot of crashes incorrectly reported as OOMs in bugsnag. Looking at breadcrumbs the available memory is around 2GB right before the app crashes. The memory usage of the app is around 70MB. This is causing a lot of noise in our reports. Is there a way for us to filter these false positives? Looks like Bugsnag needs better heuristics to report a crash as OOM

Heres some breadcrumb , App and device info at the time of the crash report

Environment

• Bugsnag version: 6.28.0
• CocoaPods version:
• Carthage version (if any):
• iOS/tvOS/macOS version(s): iOS 16
• Simulator or physical device: Physical device
• Xcode version: 15.4
• Swift version (if applicable):

[hannah-smartbear]

Hi @prabhuamol

Out of Memory errors on iOS cannot be detected directly by BugSnag, as Apple doesn't provide an event hook for these, so we use a heuristic. We monitor for the app closing unexpectedly and infer that it's an out of memory error by ruling out other known app termination causes. We currently have an item on our product roadmap to add more advanced diagnostics to Out of Memory errors. While we don’t have an ETA for when this might be implemented, you can track any updates on this on GitHub issue #1145

If you are able to determine when a crash is not an OOM based on it’s breadcrumbs, one workaround here is that you can use a callback to check this metadata. Then you could conditionally change the error message, or not report the error at all by returning false in the callback, if it is not believed to be an OOM.

Please do let us know if you have any further questions about this.

[prabhuamol]

Hi @prabhuamol

Out of Memory errors on iOS cannot be detected directly by BugSnag, as Apple doesn't provide an event hook for these, so we use a heuristic. We monitor for the app closing unexpectedly and infer that it's an out of memory error by ruling out other known app termination causes. We currently have an item on our product roadmap to add more advanced diagnostics to Out of Memory errors. While we don’t have an ETA for when this might be implemented, you can track any updates on this on GitHub issue #1145

If you are able to determine when a crash is not an OOM based on it’s breadcrumbs, one workaround here is that you can use a callback to check this metadata. Then you could conditionally change the error message, or not report the error at all by returning false in the callback, if it is not believed to be an OOM.

Please do let us know if you have any further questions about this.

Thanks Hannah. The issue you referenced has not received any updated since Sep 15, 2022. What exactly is the possible timeline here for a fix?

[prabhuamol]

@hannah-smartbear So i looked at it more and we do get a "Memory Warning" state right before the crash when the app is actually running out of memory. This state is not present for "false" OOM crashes. I wonder we can use that to categorize these errors differently. Somthing like below

Do the following in addOnSendErrorBlock

 if([[event.errors firstObject].errorClass isEqualToString:@"Out Of Memory"]) { 
 BOOL memoryWarningReceived = NO;
 for (BugsnagBreadcrumb *breadCrumb in event.breadcrumbs) {
       if (breadCrumb.type == BSGBreadcrumbTypeState && [breadCrumb.message isEqualToString:@"Memory Warning"]) {
           memoryWarningReceived = YES;
          break;
          }
 }
   if (!memoryWarningReceived) {
          event.errors[0].errorClass = @"Unknown error";
   }

     return YES;
}

[hannah-smartbear]

Hi @prabhuamol,

If you find that the UIApplicationDidReceiveMemoryWarningNotification breadcrumb being present is an accurate way to determine that the crash is actually an OOM in your case, then yes iterating through the breadcrumbs and changing the error class if the memory warning is present is a good workaround.

However please be aware that our engineers have found the heuristic that we currently use to be more accurate than low memory warnings in determining OOMs, as:

• The low memory warning doesn’t necessarily mean that the device was actually low on memory. iOS treats memory very fluidly in order to keep things running smoothly, and as such it could give a low memory warning when it's simply being used for caching, for example.

• That breadcrumb may not be present in the case of some OOMs, for example if there wasn’t enough free disk space to write the breadcrumb

Unfortunately we don’t have an ETA that we can give as to when we will make and release updates to our OOM reporting.

[prabhuamol]

Hi @prabhuamol,

If you find that the UIApplicationDidReceiveMemoryWarningNotification breadcrumb being present is an accurate way to determine that the crash is actually an OOM in your case, then yes iterating through the breadcrumbs and changing the error class if the memory warning is present is a good workaround.

However please be aware that our engineers have found the heuristic that we currently use to be more accurate than low memory warnings in determining OOMs, as:

• The low memory warning doesn’t necessarily mean that the device was actually low on memory. iOS treats memory very fluidly in order to keep things running smoothly, and as such it could give a low memory warning when it's simply being used for caching, for example.

• That breadcrumb may not be present in the case of some OOMs, for example if there wasn’t enough free disk space to write the breadcrumb

Unfortunately we don’t have an ETA that we can give as to when we will make and release updates to our OOM reporting.

Thanks @hannah-smartbear will take that into account. Unfortunately we have seen quite a few OOMs that are not related to App memory consumption. For example the app only consuming 100MB but the device is OOM and the app is killed. So trying to see if we can at least detect that case. I also submitted https://github.com/bugsnag/bugsnag-cocoa/issues/1691 Which could help a lot as we can check free memory for the app and have some threshold. < 1mb for example and consider that as an OOM. We do have that information for the device but not the App. And it's definitely available but just not exposed via an API. I hope you can get that request in.

[prabhuamol]

We can then use a combination of low memory warnings and free memory information for the app to detect an OOM vs device OOM. Maybe more reliable

[Kuniwak]

@hannah-smartbear

Out of Memory errors on iOS cannot be detected directly by BugSnag, as Apple doesn’t provide an event hook for these, so we use a heuristic.

Would it be possible to use Jetsam Event Reports? https://developer.apple.com/documentation/xcode/identifying-high-memory-use-with-jetsam-event-reports