Aggregate statistics by signature field

littleguy77 commented 11 years ago

Been working with acralyzer a fair bit today and have to say I'm loving the potential of this front-end. Thanks again for quickly resolving some db issues this weekend.

I'm guessing you have a lot of enhancements already planned, but I thought I'd chip in my two cents. At this moment the only significant thing Bugsense has over Acralyzer IMO is its aggregation of crash reports by the SIGNATURE field. I know I can get similar functionality by filtering manually in the reports page of acralyzer, but it would be nice to see the aggregate statistics at a glance for a particular error signature. Would be nice to know how many crashes have involved FooException on line XX of bar.java. Icing on the cake would be a special view showing only errors of a particular signature, with an area for devs to post comments on that error class, and the ability to mark that whole signature as resolved (and thus hidden). But now I'm getting ahead of myself. :)

Keep up the great work, and if you need a guinea pig, I'm happy to help!

william-ferguson-au commented 11 years ago

I agree. being able to aggregate based on stacktrace is the #1 feature IMHO. But the stacktrace signature will need to exclude any Exception message otherwise it will be impossible to aggregate when the message contains runtime context. It should probably use the stacktrace from the innermost (root) Exception too.

KevinGaudin commented 11 years ago

I'm happy that you feel some potential for Acralyzer :-)

I sure have some ideas for features and this is one of them. Might not be the simplest to implement depending on the number of stats you want to display.

I will work on it as soon as I finalize the background polling (and I'm nearly done, just a couple things to check when switching from one acra-storage to another).

Before starting, I would really like to get your opinions about:

what IS a signature: I have computed one for the moment but I would like to be sure that I use the right concepts. Signature is generated on the reception of the report and updating it on older reports would require to code a migration tool (or accept that older reports do not get the new signature). The comment from @william-ferguson-au is pointing in the right direction. Do you agree with it ? Do you have other ideas about what fields should be included in the signature ? I feel like the Application Version Code should be considered too. A same stack trace digest (root exception qualified name + class qualified name + line) for 2 different application versions might be a different bug because the line of code might not be the same.
what aggregate stats do you need: number of reports - ok, something else ?

I agree that a Reports Browser option to display reports sorted and/or aggregated by signature.

william-ferguson-au commented 11 years ago

+1 to including AppVersionCode in the signature.

Re aggregate reports. Number of instances and DateTimeLastReported would be sufficient for me, but it needs to be recalculated if your filters change (one of the big failings of BugSense).

littleguy77 commented 11 years ago

Honestly, just aggregating on the SIGNATURE.digest as currently computed would go a long way for me.

But to be more specific, an ideal error signature is a key built from

the classname of the inner exception and
the filename and line number from the first "interesting" element of the stack trace
- or the filename and line number from the top of the stack trace if no "interesting" element exists

I define an "interesting" element as one that doesn't involve a standard file (i.e. involves custom app code). Some examples:

Top of stack trace is custom code:

java.lang.NullPointerException
at paulscode.android.mupen64plus.free.input.map.VisibleTouchMap.drawFps(VisibleTouchMap.java:244)
at paulscode.android.mupen64plus.free.GameOverlay.onDraw(GameOverlay.java:125)
at android.view.View.draw(View.java:11094)
at android.view.View.getDisplayList(View.java:10533)
at android.view.ViewGroup.dispatchGetDisplayList(ViewGroup.java:2597)
...

The key would be generated from { "java.lang.NullPointerException", "VisibleTouchMap.java", "244" }

Top of stack trace is "standard" code, but "custom" code is in the trace:

android.view.InflateException: Binary XML file line #7: Error inflating class <unknown>
at android.view.LayoutInflater.createView(LayoutInflater.java:613)
at com.android.internal.policy.impl.PhoneLayoutInflater.onCreateView(PhoneLayoutInflater.java:56)
at android.view.LayoutInflater.onCreateView(LayoutInflater.java:660)
at android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:685)
at android.view.LayoutInflater.rInflate(LayoutInflater.java:746)
at android.view.LayoutInflater.inflate(LayoutInflater.java:489)
at android.view.LayoutInflater.inflate(LayoutInflater.java:396)
at android.view.LayoutInflater.inflate(LayoutInflater.java:352)
at android.preference.DialogPreference.onCreateDialogView(DialogPreference.java:350)
at android.preference.DialogPreference.showDialog(DialogPreference.java:288)
at paulscode.android.mupen64plus.persistent.PlayerMapPreference.show(PlayerMapPreference.java:76)
at paulscode.android.mupen64plus.PlayMenuActivity.launchGame(PlayMenuActivity.java:317)
at paulscode.android.mupen64plus.PlayMenuActivity.onPreferenceClick(PlayMenuActivity.java:168)
at android.preference.Preference.performClick(Preference.java:951)
at android.preference.PreferenceScreen.onItemClick(PreferenceScreen.java:229)
at android.widget.AdapterView.performItemClick(AdapterView.java:301)
...

The key would be generated from { "android.view.InflateException", "PlayerMapPreference.java", "76" }

And finally an example where there is no "custom" code in the trace:

android.content.ActivityNotFoundException: No Activity found to handle Intent { act=.PlayMenuActivity }
at android.app.Instrumentation.checkStartActivityResult(Instrumentation.java:1512)
at android.app.Instrumentation.execStartActivity(Instrumentation.java:1384)
at android.app.Activity.startActivityForResult(Activity.java:3190)
at android.app.Activity.startActivity(Activity.java:3297)
at android.preference.Preference.performClick(Preference.java:957)
at android.preference.PreferenceScreen.onItemClick(PreferenceScreen.java:202)
...

The key would be generated from { "android.content.ActivityNotFoundException", "Instrumentation.java", "1512" }

littleguy77 commented 11 years ago

To elaborate further, I agree that the exception message should not be part of the the key, as it's not necessary and would unnecessarily complicate the aggregation.

If we use the keys I mention above, then I don't see an obvious need for including the app version in the key. If some utility that I wrote is throwing an exception, I don't care what version of my app it came from, I just want to find it and fix it. If the front-end for the aggregation statistics allows me to narrow down the view by app ver, android ver, device model, etc. then I would think that would be plenty sufficient.

Edit: For the aggregate statistics themselves, "count" and "last seen" would be the important ones. Nice to haves would include "app versions affected", "android versions affected", and perhaps "devices affected".

william-ferguson-au commented 11 years ago

Including app version means that if you mark it as resolved (ie you fixed it) and it reoccurs in a newer version then it appears as a new bug. For bugs that re part of your app knowing that it has not been fixed is a good thing. If it is a system level bug then it could get tedious.

Instead of mining the stacktrace trying to look for a significant portion to include in the signature why wouldn't you include a hash of the whole stacktrace?

littleguy77 commented 11 years ago

Ah, good point regarding app version. I agree with that argument. I would definitely not want to hide crashes that I falsely thought were fixed.

Re: mining the stack trace. I may have belabored it a bit, and looking into the source today I realized that the mining algorithm is already implemented in acralyzer. It just searches for the first line of the trace that contains the package name and defaults to the first line if it doesn't find the package name. The only difference between what's currently implemented and what I requested is that the current implementation includes the exception message in the key.

Regarding the hash, that might be ok but it does run the risk of pulling in extra information that may be irrelevant to the developer's notion of a "signature". This could lead to situations where a single error class in the eyes of the developer are reported as multiple different errors. As an example, we have two similar apps, one is free and one is a donation version. They share mostly the same code, but they have different package names. Thus if both apps fail in the same way, from the same line of our code, the full-text hash will be different because the package names are in the full-text of the stack trace. Also, the exception message is included in the full-text, which you've already mentioned should not be incorporated into the key.

With all that said, it wouldn't be a big deal if I had to mark a bug resolved in multiple places.

KevinGaudin commented 11 years ago

A hash of the whole stack trace would vary randomly with the exception message.

I remember Jan Berkel (Soundcloud) talking about fuzzy hashing to generate hashes of stack traces that would vary only slightly if there are few differences. I've never seen an implementation though.

I agree with William, app version is necessary for marking a signature as resolve. It may reappear in a later version and you don't want to miss it. Le 19 févr. 2013 23:20, "william-ferguson-au" notifications@github.com a écrit :

Including app version means that if you mark it as resolved (ie you fixed it) and it reoccurs in a newer version then it appears as a new bug. For bugs that re part of your app knowing that it has not been fixed is a good thing. If it is a system level bug then it could get tedious.

Instead of mining the stacktrace trying to look for a significant portion to include in the signature why wouldn't you include a hash of the whole stacktrace?

— Reply to this email directly or view it on GitHubhttps://github.com/ACRA/acralyzer/issues/2#issuecomment-13803583.

littleguy77 commented 11 years ago

I think we cross-posted, but looks like we're all in agreement then about app version in the key.

william-ferguson-au commented 11 years ago

with respect using using the stacktrace, I am talking about using just that, the Java stacktrace, not exception.printstacktrace() output which also includes exception messages. Ie just class/method and line numbers.

@littleguy77 in your example: 1) Since they have different code would you really have both apps share the one Acralyzer instance? 2) Why do they have different package names? You only need to have unique Application packages. But all contained components (Activities/Services/classes) can have their own packages.

littleguy77 commented 11 years ago

1) Yes, I would like to share the acralyzer instance between both apps, since they're essentially identical (one is just a donation version but doesn't really add any functionality). 2) They have different package names in the manifest. I believe this is a requirement for publishing in the google market. I don't know all the details on that topic since I am just a contributor to the project (not the publisher).

Edit: I wasn't aware that you can change just the application package name without changing the package name everywhere. Thanks for the heads up, I'll mention it to the other devs on my project. Right now they just use a sed script to search and replace the entire code base right before publishing.

william-ferguson-au commented 11 years ago

As I said you don't need to have the same package for your Application (required to be unique for Google Play) as your other components. So your Activities and other code can be shared without having to rebadge them.

And if you did copy and rename your classes surely you would want them treated differently to the original since you have now introduced transcription and synchronisation errors.

littleguy77 commented 11 years ago

One way to have your cake and eat it too would be to exclude the app version code from the signature, and do a select operation to get all documents with a given signature where document_app_ver_code >= ver_code_when_it_was_supposedly_fixed. If you wanted to permanently hide a particular crash signature (e.g. it emanates from a closed source library) you could just set the ver_code_when_it_was_supposedly_fixed to a large number. I come from the SQL world, so I don't know how efficient or appropriate that would be for couchdb.

littleguy77 commented 11 years ago

And if you did copy and rename your classes surely you would want them treated differently to the original since you have now introduced transcription and synchronisation errors.

I'm not sure what this means exactly, but that's OK. I'll discuss with my team about not renaming the package everywhere. Also, I might be mis-representing why the publisher chose to do it that way, it's been a while since we discussed that topic. There may be another reason he chose to rename everywhere and I'm just forgetting right now.

littleguy77 commented 11 years ago

@william-ferguson-au - Thanks much for the tip regarding app package names. We had tried this earlier and failed because we were just manually changing the name in the manifest. Your discussion prompted me to look into it harder and I realized that eclipse can automatically refactor the name and will update all the dependencies correctly with a single right click on the project. Don't know how I missed that, was right under my nose the whole time.. That solved everything regarding package name. Thanks!

william-ferguson-au commented 11 years ago

Glad to be of help :-)

littleguy77 commented 11 years ago

with respect using using the stacktrace, I am talking about using just that, the Java stacktrace, not exception.printstacktrace() output which also includes exception messages. Ie just class/method and line numbers.

That works for me. Hash together each StackTraceElement along with the app version. Whether to use the app version string or the code (or both) I guess that's another decision. I would lean slightly towards the string, as I like to append "debug" to it when I'm testing between releases.

antoche commented 11 years ago

I'm just started trying Acralyzer and it's indeed great, at least as good as a lot of commercial offers! The main thing missing so far for me would also be aggregation. I was just thinking that it would be great to group issues by stacktrace and app version number, so I think we're all on par here. ;)

Note that if the purpose of putting the app version in the key is so that an error marked as fixed but reappearing in a new version gets properly flagged, there are other possible solutions. For example, the 'fixed' field could be a version number instead of a boolean. This way, exceptions can be grouped by stacktrace only, but if the same stacktrace appears in a version that's newer that the 'fix' version, then 're-open' the issue. I mainly depends on how much of a bug tracker you want Acralyzer to be.

antoche commented 11 years ago

By the way, I recommend having a look at BugSnag for the UI of navigating occurences within the same group. They have a prev/next links for each of the report subsections.

antoche commented 11 years ago

Worth a read: http://support.hockeyapp.net/kb/how-tos-faq/how-does-hockeyapp-group-crashes

KevinGaudin commented 11 years ago

Interesting, they include the 'reason' in the grouping. I assume this is the exception message ?

antoche commented 11 years ago

They actually don't define it anywhere, but from what I can see, it is the first line of the stack trace, i.e., the result of the exception's toString() method (e.g., java.lang.RuntimeException: test message).

In practice, what's sent to HockeyApp's is a big log with some specific information at the top (device info etc.), then the stacktrace, then everything else. The HockeyApp server just parses it and expects things to be in particular places. When using it with ACRA, you have full control over what's sent, so you could replace this exception line with "Rubber duckies" if you wished so, and HockeyApp would pick that as the "reason".

KevinGaudin commented 11 years ago

The most difficult part may be to layout the data: acra-bugs

antoche commented 11 years ago

The most practicals UI let you open a given bug and navigate through each of the reports that caused that particular bug. To better layout and display the bugs, you can add a lot of functionality such as:

Assign bugs priority or sort by more recent report first,
Mark bugs as 'resolved' so the bugs disappear from the list even if new reports come in (as discussed above, it needs to be smart enough that reports for a version in which the bug was resolved would not be ignored)
Filter bugs by app version, OS version, etc

The more features you add and the more it becomes a bug tracker. You probably want to draw the line at some point and look into integration with bug trackers rather than try to do everything. I think the most logical split would be:

Everything that only Acralyzer knows (i.e., the aggregation logic) should be done in Acralyzer
All the rest should be left to the big tracker. That includes the issue list itself. Acralyzer can just keep a pointer to the issue URL or id.

The difficult part lies in the data that's shared by both. E.g., if storing a '"fixed version" associated with the bug to know whether to aggregate an incoming report or create a new bug from it, would that "fixed version" field be stored in the bug tracker (ideal, but Acralyzer needs to be able to access it), in Acralyzer only (easier, but it's useful data for a bug tracker too), or both (and risk that they mismatch)?

With all that in mind, some minimum bug handling in Acralyzer like the one in your screenshot is probably quite practical for people who don't want to bother with a bug tracker.

KevinGaudin commented 11 years ago

Here's the state of my development after a few hours today: https://www.youtube.com/watch?v=50C9D_rPXmg

What you can see in that video (sorry about the mouse cursor not showing, this is how VLC records a desktop):

There is a new "Bugs" tab. A bug is a reports aggregate. It is defined by an App version code, Exception and Root Exception. The whole exception line is used for the moment, including the message.
The bugs list shows for each bug:
- the moment of the latest report
- the number of reports
- the Exception and root Exception
There is a button to show/hide solved bugs
Clicking on the resolution state (first column) toggles its solved/unsolved state.

That's it for today. Nothing changed anywhere else.

Please note that the list update animations are crappy because:

I just copy/pasted them from elsewhere without further tweaking
the list goes through a full update when a bug resolution state changed. This is because the background polling is detecting a change in the database and reloads everything. This will have to be tweaked too.

You can try it right now, you'll need to update your acra-storage couchapp (still compatible with previous versions of acralyzer) and get acralyzer from the aggregates branch. Be aware that I can't promise that the resolution state storage used in this version will be preserved in further releases. Everything here is experimental until merged on the master branch.

I agree with @antoche about the line to be drawn in terms of features. My opinion is that Acralyzer has to focus on helping to identify, understand and solve bugs. It's all about collecting reports, browsing/searching through them, understanding them.

antoche commented 11 years ago

Very promising! Will try it ASAP!

On Thu, May 2, 2013 at 9:08 AM, Kevin Gaudin notifications@github.comwrote:

Here's the state of my development after a few hours today: https://www.youtube.com/watch?v=50C9D_rPXmg

What you can see in that video (sorry about the mouse cursor not showing, this is how VLC records a desktop):

There is a new "Bugs" tab. A bug is a reports aggregate. It is defined by an App version code, Exception and Root Exception. The whole exception line is used for the moment, including the message.

The bugs list shows for each bug:

the moment of the latest report

the number of reports

the Exception and root Exception

There is a button to show/hide solved bugs

Clicking on the resolution state (first column) toggles its solved/unsolved state.

That's it for today. Nothing changed anywhere else.

Please note that the list update animations are crappy because:

I just copy/pasted them from elsewhere without further tweaking

the list goes through a full update when a bug resolution state changed. This is because the background polling is detecting a change in the database and reloads everything. This will have to be tweaked too.

You can try it right now, you'll need to update your acra-storage couchapp (still compatible with previous versions of acralyzer) and get acralyzer from the aggregates branchhttps://github.com/ACRA/acralyzer/tree/aggregates. Be aware that I can't promise that the resolution state storage used in this version will be preserved in further releases. Everything here is experimental until merged on the master branch.

I agree with @antoche https://github.com/antoche about the line to be drawn in terms of features. My opinion is that Acralyzer has to focus on helping to identify, understand and solve bugs. It's all about collecting reports, browsing/searching through them, understanding them.

— Reply to this email directly or view it on GitHubhttps://github.com/ACRA/acralyzer/issues/2#issuecomment-17306580 .

johnuopini commented 11 years ago

Works! Only issue i see is when "background polling" is not active, in that case clicking on the status won't change anything visually or reload the page (but it will change the state on the db, so after manual a reload status will be toggled).

KevinGaudin commented 11 years ago

It's starting to look good: https://www.youtube.com/watch?v=UEGfb1CErqQ

You can see in this screencast:

The new Dashboard, with a new "Bugs" tab showing the latest unsolved received bugs and the 'old' Reports tab, showing all latest received reports
Lists of bugs display the moment of the latest report, number of reports, application version code, exception and root exception for each bug
You can mark a bug as solved by clicking on the small bug icon. There is a switch to show solved bugs.
Clicking on the magnifier icon leads you to the details of this bug, with a list of all its reports. You can switch the resolution state on this page too, but also enter a description for this bug.
The description can use Markdown for styling.
Back on the bugs list, when a bug has a description, the magnifier icon is dotted. When your mouse goes over the magnifier, a tooltip is displayed with the styled description.

paour commented 11 years ago

The current implementation is a great improvement, and I've started using it in production, it's really nice and fast.

I wish that when clicking on a bug in the list of aggregated bugs (bugs-browser) I didn't have to click again on one of the reports (in the reports-browser view) to get the stack trace. You could either include the stack trace in the Bug Summary or pre-select the latest report in the list.

littleguy77 commented 11 years ago

Wow, this is really fantastic progress!! I briefly checked out the aggregates branch a while back but decided not to push it out to our db since it was still early going. Just updated and merged the aggregates branch today and it looks to provide pretty much everything we need, without a whole lot of fuss.

If you're looking for a few more suggestions, some icing on the cake would be

In bugs-table, allow the list to be ordered by number of reports (descending)
In dashboard, add a pie chart of bug/signature (ideally omitting hidden bugs)

As I said, icing. As it stands, it pretty much gives us everything I can think of. Awesome work! :D

KevinGaudin commented 11 years ago

Thank you guys for testing, I'm happy that you like it. There are still a few things I'd like to do before merging on the master branch, suggestions for icing on the cake are welcome ;-)

KevinGaudin commented 11 years ago

The aggregates branch has been updated with my own in-browser monsterid generator + ability to filter reports by user while examining a bug.

You can now also get a list of all reports from one user by clicking on its monster in the reports details summary table.

The reports summary layout has been slightly rearranged to give more air to stack trace and custom data. (inspired by @littleguy77) ;-)

littleguy77 commented 11 years ago

Thanks Kevin, this is great. If you want to see some "real data" as of we have over 3000 reports collected in the past 10 days. https://paulscode.iriscouch.com/acralyzer-experimental/_design/acralyzer/index.html#/dashboard/mupen64plusae https://paulscode.iriscouch.com/acralyzer-custom/_design/acralyzer/index.html#/dashboard/mupen64plusae

The first is from your aggregates branch, the second has some of my customizations on top. If you forget the login credentials, just shoot me an email.

Basically the biggest thing missing now is the ability to sort bugs by number of reports. Latest report is mostly irrelevant to us.

KevinGaudin commented 11 years ago

With the latest version that I just merged on the master branch, bugs are now sorted by number or reports on the 'Bugs' tab (not the dashboard). On this tab, you can also sort by latest report and application version code. These 3 ordering can be switched ascending/descending. Just click on the column headers.

KevinGaudin commented 11 years ago

I'm closing this thread as I think the main basic set of features is now implemented.

If you want to go further in aggregates analysis, please open new issues.

ACRA / acralyzer

Aggregate statistics by signature field #2