Crash dump feature for client libraries

[mattheworiordan]

Background

When customers experience a crash and report this to the Ably support team, it's considerably more difficult for us to help them without verbose logs. As such, we often tell customers to enable verbose logging and send us the logs, but this presents a few issues:

• It's after the fact, so typically too late and often hard to reproduce
• Customers need to implement their own custom loggers often to capture verbose logs but not output the logs to STDOUT / console, manage how many logs they capture (memory conservation), and then implement some mechanism to dump those logs at the time of a failure.
• The verbose logs may contain personally identifiable information
• If the SDKs are part of an app store, the time it takes them to roll out a change like this is prohibitive.
• We're asking developers to implement common functionality that is hard to do properly, and thus means their DX is diminished, and we're less likely to be able to help them.

Somewhat related to this, we added (optional) functionality to the spec to that allows exceptions to be caught and sent to our own exception tracking service. The purpose of this was also to help us identify issues customers are having and get fixes out more quickly.

We also added a spec item to handle failures as gracefully as possible, see https://docs.ably.io/client-lib-development-guide/features/#RSC20a.

Proposal

I propose we consider adding the following functionality to the libraries, which if agreed, I can update the spec for review:

• Where unexpected exceptions are handled gracefully, we should report the exception along with an instruction in the log recommending they capture the crash dump from the Ably client in order to get help from Ably. The only tricky thing here is because of our spec requirement to gracefully handle exceptions, do we unilaterally go with this log message, or only for internal client 50x style errors? Do we have a set of error codes reserved for internal exceptions? If not, we should define some and hope that progressively over time we will emit error codes that clearly indicate whether this is service/network fault, a 3rd party code fault (such as an exception in a callback), or an SDK fault.
• We expose a method such as client.util.getCrashDump() which returns a plain text formatted string constructed from:
  • Last ClientOptions.log.crashDumpBufferLength bytes (perhaps default 32KiB) from verbose logs. Note that the standard log level has no impact on the getCrashDump log level. ClientOptions.log.crashDumpLogLevel will default to the most verbose level
  • Message data only will be obfuscated if ClientOptions.log.crashDumpObfuscate is true (default true)
• If ClientOptions.log.crashDumpEnabled is true (default false in 1.* at least), then an exception is raised when calling this method.
• Encourage everyone to consider if RSC20a is in fact implemented correctly.
• If the client lib supports exception reporting to Ably, then we should bundle in the response from getCrashDump at the time of sending the exception.

┆Issue is synchronized with this Jira Task by Unito

[mattheworiordan]

Assigned @SimonWoolf and @paddybyers for review

[ably-sync-bot]

➤ Simon Woolf commented:

this is an issue for quintin not me, reassigning

[sync-by-unito[bot]]

➤ Automation for Jira commented:

The link to the corresponding Jira issue is https://ably.atlassian.net/browse/SDK-2805