Integrate crash reporting tools

[NaterGator]

At least on non-mobile applications we would be well served to handle crashes more gracefully. As mentioned here a decent choice would be the Chromium project's breakpad library.

We can start small and just generate crash dumps when an unhandled exception occurs (users would have to manually upload them). Later down the line it would be nice to have a built in console / tool to facilitate users submitting stored crash reports.

[NaterGator]

@DonLakeFlyer what do you think about integration with https://drdump.com ? We can upload the release build symbols to their service and collect crash dumps from Windows there.

[DonLakeFlyer]

That sound like a great idea. Do we need to start flipping some bits in the installer to get better crash reports?

[jaxxzer]

@NaterGator have you done any work on this? I'm interested in helping get this together.

[NaterGator]

I looked into this and we would have to package binary blobs that we can't audit from a software company in Russia. Didn't / doesn't feel right. I'd like to find an open-source and auditable solution to this problem.

[DonLakeFlyer]

Breakpad support is working it's way into 3.4 release hopefully

[patrickelectric]

Hey guys, any update on it ? probably waiting for 5.12 to have the debug symbols ? Any motive to use breakpad and not other project ?

[DonLakeFlyer]

Nobody is working on this. Not a small amount of work.

[patrickelectric]

I'm adding a crash report for one application, you can see the output window here:

After doing the deploy and tests for windows , linux and mac. I'll add it in QGC.

[DonLakeFlyer]

Would be fantastic to get this all working cross-platform. Huge win.

[jaxxzer]

We are doing some work with KCrash and drkonqi

We do intend to attempt some integration with QGC. Will let you all know how that goes.

[DonLakeFlyer]

The listed support for KCrash is "FreeBSD, Linux, Windows". Does that include OSX and Android?

[patrickelectric]

The only unsupported OS is Android for now, I did some work and tests with this patch: https://phabricator.kde.org/D4929, and we got it merged last year for OSX. Android will be supported soon since KDE is strongly working to improve plasma mobile support and kirigami applications.

[DonLakeFlyer]

The thing is that we already have a working crash system for Android using breakpad. It was funded by a commercial partner. We just need to get permission to pull it down to open source from a private repo. And breakpad is supposed to work on all platforms. If you want I can ask permission to bring that into the main repo.

[patrickelectric]

Well, if that is possible, I would like very much to take a look on it and to help with the others OS integration in QGC :)

[DonLakeFlyer]

Ok, let me ask for permission.

[jaxxzer]

If you want I can ask permission to bring that into the main repo.

Please do. This is exactly why I bring up our own work here: In order to align efforts.

[jaxxzer]

We are continuing to work with integrating kcrash with qgc. @tcanabrava can provide more details about that. @DonLakeFlyer please let us know if you can share the breakpad + android project. In any event, we can use kcrash for the other platforms. Problems on Windows machines have been my main concern. We've had good luck everywhere else.

[DonLakeFlyer]

please let us know if you can share the breakpad + android project

Sorry. I never got permission for that.

What exactly does kcrash provide as far as features? FYI: The Windows builds already have crash dump capture turned on for them by the installer. Which you can then load into VS.

[jaxxzer]

What exactly does kcrash provide as far as features?

@tcanabrava can help to explain it better, but as I understand it, kcrash provides the facility to collect the stack trace, and drkonqi is an optional program that can be installed independently of qgc and can provide the 'program crashed' ui that presents the stack trace to the user and gives them the option to report it.

FYI: The Windows builds already have crash dump capture turned on for them by the installer. Which you can then load into VS.

Thank you for the heads up. I have used this process before. I found it excessively difficult and time consuming to get right with the .pdbs from S3. The crash dump files are also huge (> hundreds of MB). These crash dumps provide a more introspective option for debugging, but the stack trace via kcrash will provide a quick and easy debugging option and facilitate identifying patterns.

[DonLakeFlyer]

The crash dump files are also huge (> hundreds of MB).

Also just FYI: I changed them to be minidumps a few weeks ago.

drkonqi is an optional program that can be installed independently of qgc and can provide the 'program crashed' ui that presents the stack trace to the user and gives them the option to report it.

Hmm. Installing another app on the machine to make this all work is a bit concering.

[jaxxzer]

Also just FYI: I changed them to be minidumps a few weeks ago.

:+1:

Hmm. Installing another app on the machine to make this all work is a bit concering.

AFAIK: kcrash is the only dependency we need to capture the stack trace. I think we could simply write that report to a file and tell the user where it is (@tcanabrava can clarify). drkonqi is an optional dependency that presents a UI with the kcrash report. drkonqi could be bundled with QGC, or installed separately to reduce bloat.

[tcanabrava]

@DonLakeFlyer KCrash + DrKonqi are a combination of thigns. KCrash is a crash retrieval feature that supports generation of backtraces in mac / linux / windows / iOS (and android is comming), it's fully opensource and used in almost all softwares from the KDE community. for your app to support KCrash, you basically call two functions after the creation of QApplication (and of course, link to the library)

DrKonqi is the front-end that displays to the user the backtrace and asks for permission to create a bug report. I started fixing bugs in it this month so it's easier integrated in non-kde projects. (this is not finished, but I hope to finish soon)

The usability goes like this: QGC Crashes DrKonqi displays the backtrace User clicks in "Create Bugreport" The backtrace + User comments are submited to the bugracker.

What I still need to do in DrKonqi for a correct integration in QGC:

• Support Github Tickets (currently it only supports bugzilla)
• Support Per-Project configuration (Currently it will send to bugs.kde.org by default)

[jaxxzer]

Support Github Tickets (currently it only supports bugzilla)

@tcanabrava Is this optional? Could an email address be used instead?

[tcanabrava]

not optional, we are working in backends for it.

[jaxxzer]

@DonLakeFlyer @tcanabrava

I recommend we forgo the drkonqi UI + automatic reporting tools right now (they require further development).

We can use kcrash only (the user only needs to install qgc), and when a crash occurs, write a stack trace to a text file (perhaps under Documents/QGC/crashlog). When a user reports a crash, we can request them to retrieve this file on a case by case basis.

[DonLakeFlyer]

I recommend we forgo the drkonqi UI + automatic reporting tools right now

Sounds fine.

I'm interesting in seeing what sort of data you get back from this though. @patrickelectric Gave an example but it is fully Qt based. For example on Windows without a PDB you aren't going to get function names for QGC code will you?

[tcanabrava]

No, a pdb is needed. But the pdb could be downloaded via the interface when the user explicitly clicked a button for it

On Fri, 20 Sep 2019 at 21:15 Don Gagne notifications@github.com wrote:

I recommend we forgo the drkonqi UI + automatic reporting tools right now

Sounds fine.

I'm interesting in seeing what sort of data you get back from this though. @patrickelectric https://github.com/patrickelectric Gave an example but it is fully Qt based. For example on Windows without a PDB you aren't going to get function names for QGC code will you?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mavlink/qgroundcontrol/issues/4445?email_source=notifications&email_token=AA6XGUCPEMTZXD5PR55QLYDQKUOKZA5CNFSM4C4S6LI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HUKOA#issuecomment-533677368, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6XGUE2ST2ESJS3QRSH4QTQKUOKZANCNFSM4C4S6LIQ .

[DonLakeFlyer]

That really makes me wonder about the usefulness of this then. For example is the effort more well spent to get QGC Windows into the store. Which will both gets us nice server based crash reporting as well as store support. Better upgrade support and so forth. Might be just as much work to do that. Kill two birds with one stone.

[tcanabrava]

I can't say, Don, as I don't use windows for almost 20 years. Does the store support on windows also helps the Mac and Linux codebases?

On Sun, Sep 22, 2019 at 9:37 PM Don Gagne notifications@github.com wrote:

That really makes me wonder about the usefulness of this then. For example is the effort more well spent to get QGC Windows into the store. Which will both gets us nice server based crash reporting as well as store support. Better upgrade support and so forth. Might be just as much work to do that. Kill two birds with one stone.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mavlink/qgroundcontrol/issues/4445?email_source=notifications&email_token=AA6XGUGE7N773WU6SKSTAODQK7COBA5CNFSM4C4S6LI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7JNNSI#issuecomment-533911241, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6XGUAVIKLX4C724ACUQ3DQK7COBANCNFSM4C4S6LIQ .

[DonLakeFlyer]

Does the store support on windows also helps the Mac and Linux codebases?

No it doesn't. But the fact is that Windows is the largest use case and also tends to be the OS which is the quirkiest. Hence information on crashes caught there will benefit all OS. It is very rare for there to be a crash which is OSX or Linux specific. So if you can get the prevalent desktop build reporting that will cover most everything.

[jaxxzer]

Solid point about the pdb and so on @DonLakeFlyer. We will look into the store options you mentioned and bring them into consideration.

[DonLakeFlyer]

The thing is we already have iOS and Android covered for crash reporting on mobile since they are in the store. We are just missing a desktop version.