search

the-themis-benchmarks / home

The Themis Benchmark for evaluating automated GUI testing
MIT License
142 stars 19 forks source link

Is there a problem of repeated counting of crashes? #4

Closed helix55 closed 2 years ago

helix55 commented 2 years ago

We triggered ActivityDiary-1.1.8-debug-#118 only once, which was captured by Runtime and ACRA respectively, so two traces with the same signature appeared in logcat. Like this:

11-04 09:43:08.822 5041 5041 E AndroidRuntime: FATAL EXCEPTION: main 11-04 09:43:08.822 5041 5041 E AndroidRuntime: Process: de.rampro.activitydiary.debug, PID: 5041 11-04 09:43:08.822 5041 5041 E AndroidRuntime: java.lang.IllegalArgumentException: position (0) too small 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at de.rampro.activitydiary.ui.generic.DetailRecyclerViewAdapter.getDiaryImageIdAt(DetailRecyclerViewAdapter.java:156) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at de.rampro.activitydiary.ui.history.HistoryRecyclerViewAdapter$1.onClick(HistoryRecyclerViewAdapter.java:87) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at android.support.v7.app.AlertController$ButtonHandler.handleMessage(AlertController.java:162) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at android.os.Handler.dispatchMessage(Handler.java:105) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at android.os.Looper.loop(Looper.java:164) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at android.app.ActivityThread.main(ActivityThread.java:6541) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at java.lang.reflect.Method.invoke(Native Method) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240) 11-04 09:43:08.822 5041 5041 E AndroidRuntime: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767) 11-04 09:43:08.824 5041 5041 E ACRA : ACRA caught a IllegalArgumentException for de.rampro.activitydiary.debug 11-04 09:43:08.824 5041 5041 E ACRA : java.lang.IllegalArgumentException: position (0) too small 11-04 09:43:08.824 5041 5041 E ACRA : at de.rampro.activitydiary.ui.generic.DetailRecyclerViewAdapter.getDiaryImageIdAt(DetailRecyclerViewAdapter.java:156) 11-04 09:43:08.824 5041 5041 E ACRA : at de.rampro.activitydiary.ui.history.HistoryRecyclerViewAdapter$1.onClick(HistoryRecyclerViewAdapter.java:87) 11-04 09:43:08.824 5041 5041 E ACRA : at android.support.v7.app.AlertController$ButtonHandler.handleMessage(AlertController.java:162) 11-04 09:43:08.824 5041 5041 E ACRA : at android.os.Handler.dispatchMessage(Handler.java:105) 11-04 09:43:08.824 5041 5041 E ACRA : at android.os.Looper.loop(Looper.java:164) 11-04 09:43:08.824 5041 5041 E ACRA : at android.app.ActivityThread.main(ActivityThread.java:6541) 11-04 09:43:08.824 5041 5041 E ACRA : at java.lang.reflect.Method.invoke(Native Method) 11-04 09:43:08.824 5041 5041 E ACRA : at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240) 11-04 09:43:08.824 5041 5041 E ACRA : at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767)

Run the check_crash.py script to count the crashes, and output 2 crashes, but in fact only one crash is triggered.

[ActivityDiary, #118] the crash was triggered (2) times [ActivityDiary, #118] the time duration: ['2', '2'] (mins)

Will this statistical result bias the assessment of the tool’s ability to find bugs?

tingsu commented 2 years ago

Oh, this is a nice catch! I think this is a real issue, which will duplicate the crash times. We will fix this issue soon.

tingsu commented 2 years ago

Root cause analysis In Themis, for each testing tool, we use logcat to filter target crashes. E.g., in monkey.sh, we filter both "AndroidRuntime:E" and "ACRA:E" via logcat: https://github.com/the-themis-benchmarks/home/blob/073129993509217612e860021b534a7f420b42f9/scripts/run_monkey.sh#L97 If an app uses ACRA (a crash reporting library, https://github.com/ACRA/acra), the app may throw the crash message (i.e., crash stack trace) twice for the same crashing bug (one time for "ACRA:E", another time for "AndroidRuntime:E"). Thus, Themis will double count number of triggered crashes. This issue does not affect the core correctness of Themis, but affects this countering.

How to fix A clear fixing way is to (1) disable the ACRA library in those apps which use ACRA, and (2) remove the filter "ACRA:E".

@Apochens , can you help fix this issue?

Apochens commented 2 years ago

I have disabled the ACRA in commons(#2123, #3244) and osmeditor4Android(#729) by commenting the line

ACRA.init(this)

It does work.

tingsu commented 2 years ago

I created a wiki for Themis to clarify such issues (helping future contributors for extending our bug dataset). See: https://github.com/the-themis-benchmarks/home/wiki/Extend-Themis-for-our-research-community

Apochens commented 2 years ago

This bug is fixed in #5 .