Closed helix55 closed 2 years ago
Oh, this is a nice catch! I think this is a real issue, which will duplicate the crash times. We will fix this issue soon.
Root cause analysis In Themis, for each testing tool, we use logcat to filter target crashes. E.g., in monkey.sh, we filter both "AndroidRuntime:E" and "ACRA:E" via logcat: https://github.com/the-themis-benchmarks/home/blob/073129993509217612e860021b534a7f420b42f9/scripts/run_monkey.sh#L97 If an app uses ACRA (a crash reporting library, https://github.com/ACRA/acra), the app may throw the crash message (i.e., crash stack trace) twice for the same crashing bug (one time for "ACRA:E", another time for "AndroidRuntime:E"). Thus, Themis will double count number of triggered crashes. This issue does not affect the core correctness of Themis, but affects this countering.
How to fix A clear fixing way is to (1) disable the ACRA library in those apps which use ACRA, and (2) remove the filter "ACRA:E".
@Apochens , can you help fix this issue?
I have disabled the ACRA in commons(#2123, #3244) and osmeditor4Android(#729) by commenting the line
ACRA.init(this)
It does work.
I created a wiki for Themis to clarify such issues (helping future contributors for extending our bug dataset). See: https://github.com/the-themis-benchmarks/home/wiki/Extend-Themis-for-our-research-community
This bug is fixed in #5 .
We triggered ActivityDiary-1.1.8-debug-#118 only once, which was captured by Runtime and ACRA respectively, so two traces with the same signature appeared in logcat. Like this:
Run the check_crash.py script to count the crashes, and output 2 crashes, but in fact only one crash is triggered.
Will this statistical result bias the assessment of the tool’s ability to find bugs?