search

dacapobench / dacapobench

The DaCapo benchmark suite
https://www.dacapobench.org/
Apache License 2.0
159 stars 60 forks source link

PMD sometimes yields NullPointerException where lint error message is expected #310

Open wks opened 2 days ago

wks commented 2 days ago

When running the pmd benchmark in dacapo-23.11-chopin.jar, there is a chance that the digest verification will fail. The generated "pmd-report.txt" differs from the expected output.

It can be reproduced by simply running the following command repeatedly.

java -jar dacapo-23.11-chopin.jar -n 1 pmd -preserve

The MMTk project uses DaCapo to test its OpenJDK binding in the CI, and it often produces error when running pmd. See: https://github.com/mmtk/mmtk-openjdk/actions/runs/11064388662/job/30742621309?pr=289

I collected three bad pmd-report.txt outputs.

Output 1 (run with MMTk-OpenJDK with OpenJDK 11.0.19):

--- pmd-report.txt.good 2024-09-27 13:46:44.293662086 +0800
+++ pmd-report.txt.bad  2024-09-27 13:46:14.930327743 +0800
@@ -993,6 +993,7 @@
 src/main/java/net/sourceforge/pmd/RuleSetLoader.java:69:       MethodArgumentCouldBeFinal:     Parameter 'loader' is not assigned and could be declared final
 src/main/java/net/sourceforge/pmd/RuleSetLoader.java:81:       MethodArgumentCouldBeFinal:     Parameter 'minimumPriority' is not assigned and could be declared final
 src/main/java/net/sourceforge/pmd/RuleSetLoader.java:92:       MethodArgumentCouldBeFinal:     Parameter 'warn' is not assigned and could be declared final
+src/main/java/net/sourceforge/pmd/RuleSetNotFoundException.java        -       NullPointerException: null
 src/main/java/net/sourceforge/pmd/RuleSetNotFoundException.java:16:    CommentRequired:        Public method and constructor comments are required
 src/main/java/net/sourceforge/pmd/RuleSetNotFoundException.java:16:    MethodArgumentCouldBeFinal:     Parameter 'msg' is not assigned and could be declared final
 src/main/java/net/sourceforge/pmd/RuleSetNotFoundException.java:20:    CommentRequired:        Public method and constructor comments are required

Output 2: (run with MMTk-OpenJDK with OpenJDK 11.0.19)

--- pmd-report.txt.good 2024-09-27 13:46:44.293662086 +0800
+++ pmd-report.txt.bad2 2024-09-27 14:15:27.717054648 +0800
@@ -195,6 +195,7 @@
 src/main/java/net/sourceforge/pmd/PMDConfiguration.java:98:    LongVariable:   Avoid excessively long variable names like DEFAULT_SUPPRESS_MARKER
 src/main/java/net/sourceforge/pmd/PMDConfiguration.java:993:   CommentSize:    Comment is too large: Line too long
 src/main/java/net/sourceforge/pmd/PMDConfiguration.java:994:   CommentSize:    Comment is too large: Line too long
+src/main/java/net/sourceforge/pmd/PMDException.java    -       NullPointerException: null
 src/main/java/net/sourceforge/pmd/PMDException.java:20:        CommentRequired:        Field comments are required
 src/main/java/net/sourceforge/pmd/PMDException.java:28:        MethodArgumentCouldBeFinal:     Parameter 'message' is not assigned and could be declared final
 src/main/java/net/sourceforge/pmd/PMDException.java:32:        CommentSize:    Comment is too large: Too many lines

Output 3: (run with openjdk 22.0.2.u9-1 from ArchLinux)

--- pmd-report.txt.good 2024-09-27 13:46:44.293662086 +0800
+++ pmd-report.txt.bad3 2024-09-27 15:05:47.637156828 +0800
@@ -8820,7 +8820,7 @@
 src/main/java/net/sourceforge/pmd/renderers/CSVWriter.java:80: MethodArgumentCouldBeFinal:     Parameter 'buffer' is not assigned and could be declared final
 src/main/java/net/sourceforge/pmd/renderers/CSVWriter.java:80: MethodArgumentCouldBeFinal:     Parameter 's' is not assigned and could be declared final
 src/main/java/net/sourceforge/pmd/renderers/CSVWriter.java:80: ShortVariable:  Avoid variables with short names like s
-src/main/java/net/sourceforge/pmd/renderers/CodeClimateIssue.java:13:  DataClass:      The class 'CodeClimateIssue' is suspected to be a Data Class (WOC=0.000%, NOPA=8, NOAM=0, WMC=1)
+src/main/java/net/sourceforge/pmd/renderers/CodeClimateIssue.java      -       NullPointerException: Cannot read field "visibility" because "sig" is null
 src/main/java/net/sourceforge/pmd/renderers/CodeClimateIssue.java:14:  CommentRequired:        Field comments are required
 src/main/java/net/sourceforge/pmd/renderers/CodeClimateIssue.java:15:  CommentRequired:        Field comments are required
 src/main/java/net/sourceforge/pmd/renderers/CodeClimateIssue.java:15:  VariableNamingConventions:      Only variables that are final should contain underscores (except for underscores in standard prefix/suffix), 'check_name' is not final.

What's common in the three runs is that the line number is not printed, and it shows NullPointerException where a lint error message should be printed.

The two erroneous outputs (1 and 2) from MMTk showed null as the message of the exception. That may be related to memory management details of MMTk or OpenJDK 11.0.19.

Output 3 showed the exception message Cannot read field "visibility" because "sig" is null. It should be caused by an expression sig.visibility where sig is null. Such expression occurs twice in PMD 6.55.0. I still don't know why this happens, yet, but I think it is either a bug in PMD 6.55.0 itself or the test harness. It should be possible to find the reason by printing the stack trace at the NullPointerException.

steveblackburn commented 2 days ago

What failure rate are you seeing? I just ran ~200~ 1100 invocations and saw no failures.

steveblackburn commented 2 days ago

My initial tests were on JDK 17.

I've now moved to JDK 11 and I can see occasional failures (8/1484). It's interesting that I saw zero failures under JDK 17. I'll do more extensive testing with JDK 17 to see whether the failure eventually surfaces.