Regression report displays exceptions inconsistently

[Lmh-java]

Discovered at https://github.com/checkstyle/checkstyle/pull/14791#issuecomment-2053817456

Regression report displays exceptions inconsistently.

Sometimes there is a full stacktrace for an exception: https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/elasticsearch/index.html#A139

Sometimes there is only a line of the exception name: https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/elasticsearch/index.html#A1

Both these two cases came from the same report generation: https://checkstyle-diff-reports.s3.us-east-2.amazonaws.com/8039a13_2024203306/reports/diff/index.html

---

To reproduce

1. Create a new branch contains the fix for https://github.com/checkstyle/checkstyle/pull/14791.
2. Create a new branch from the commit before https://github.com/checkstyle/checkstyle/pull/14791 is merged.
3. Prepare the following configuration: Diff Regression config: https://gist.githubusercontent.com/Lmh-java/95ba8a3850cf6d670b2d65262531fb35/raw/7f17357bf8ec422286181c2d470ed578f8da485d/magic-number-config.xml project lists: use the default (or only the following line for bare minimum)

   elasticsearch|git|https://github.com/elastic/elasticsearch|v1.5.2||

4. run

   groovy diff.groovy --localGitRepo [location of checkstyle project] --baseBranch [branch name without fix] --patchBranch  [branch name with fix] --config magic-number-config.xml --listOfProjects test.properties -xm -Dcheckstyle.failsOnError=false --allowExcludes

Some analysis

After investigation, this difference presents before the results are sent to patch-diff-report-tool to compare. This difference is presented in checkstyle-result.xml. For example:

<file name="/Users/liminghao/Programs/IdeaProjects/contribution/checkstyle-tester/repositories/elasticsearch/src/test/java/org/elasticsearch/bwcompat/UnicastBackwardsCompatibilityTest.java">
<error line="1" severity="error" message="Got an exception - java.lang.NullPointerException: Cannot invoke &quot;com.puppycrawl.tools.checkstyle.api.DetailAST.getType()&quot; because &quot;node&quot; is null&#10;#x9;at com.puppycrawl.tools.checkstyle.checks.coding.MagicNumberCheck.isFieldDeclaration(MagicNumberCheck.java:390)&#10;#x9;at com.puppycrawl.tools.checkstyle.checks.coding.MagicNumberCheck.visitToken(MagicNumberCheck.java:219)&#10;#x9;at com.puppycrawl.tools.checkstyle.TreeWalker.notifyVisit(TreeWalker.java:335)&#10;#x9;at com.puppycrawl.tools.checkstyle.TreeWalker.processIter(TreeWalker.java:408)&#10;#x9;at com.puppycrawl.tools.checkstyle.TreeWalker.walk(TreeWalker.java:273)&#10;#x9;at com.puppycrawl.tools.checkstyle.TreeWalker.processFiltered(TreeWalker.java:154)&#10;#x9;at com.puppycrawl.tools.checkstyle.api.AbstractFileSetCheck.process(AbstractFileSetCheck.java:101)&#10;#x9;at com.puppycrawl.tools.checkstyle.Checker.processFile(Checker.java:340)&#10;#x9;at com.puppycrawl.tools.checkstyle.Checker.processFiles(Checker.java:299)&#10;#x9;at com.puppycrawl.tools.checkstyle.Checker.process(Checker.java:226)&#10;#x9;at org.apache.maven.plugins.checkstyle.exec.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:202)&#10;#x9;at org.apache.maven.plugins.checkstyle.AbstractCheckstyleReport.executeReport(AbstractCheckstyleReport.java:533)&#10;#x9;at org.apache.maven.plugins.checkstyle.CheckstyleReport.executeReport(CheckstyleReport.java:57)&#10;#x9;at org.apache.maven.reporting.AbstractMavenReport.generate(AbstractMavenReport.java:255)&#10;#x9;at org.apache.maven.plugins.site.render.ReportDocumentRenderer.renderDocument(ReportDocumentRenderer.java:226)&#10;#x9;at org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.render(DefaultSiteRenderer.java:348)&#10;#x9;at org.apache.maven.plugins.site.render.SiteMojo.renderLocale(SiteMojo.java:194)&#10;#x9;at org.apache.maven.plugins.site.render.SiteMojo.execute(SiteMojo.java:143)&#10;#x9;at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)&#10;#x9;at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)&#10;#x9;at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)&#10;#x9;at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)&#10;#x9;at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)&#10;#x9;at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)&#10;#x9;at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)&#10;#x9;at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)&#10;#x9;at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)&#10;#x9;at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)&#10;#x9;at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)&#10;#x9;at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)&#10;#x9;at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)&#10;#x9;at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)&#10;#x9;at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)&#10;#x9;at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)&#10;#x9;at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906)&#10;#x9;at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)&#10;#x9;at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)&#10;#x9;at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)&#10;#x9;at java.base/java.lang.reflect.Method.invoke(Method.java:580)&#10;#x9;at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283)&#10;#x9;at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226)&#10;#x9;at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)&#10;#x9;at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)&#10;" source="com.puppycrawl.tools.checkstyle.Checker"/>
</file>

<file name="/Users/liminghao/Programs/IdeaProjects/contribution/checkstyle-tester/repositories/elasticsearch/src/test/java/org/elasticsearch/script/ScriptFieldTests.java">
<error line="1" severity="error" message="Got an exception - java.lang.NullPointerException&#10;" source="com.puppycrawl.tools.checkstyle.Checker"/>
</file>

This means, when the checkstyle report is generated, this difference is already there. I also noticed that the single line stacktrace has &#10(\n) at the very end. This might be the cause of the incorrect termination of the line.

[nrmancuso]

Please share a reliable project/ config combination to reproduce this with.

[Lmh-java]

Please share a reliable project/ config combination to reproduce this with.

Done, updated in the issue description. Please see whether that's sufficient to reproduce.

[nrmancuso]

Looks good, thank you @Lmh-java