search

ScaCap / action-surefire-report

Reports surefire test results as GitHub Pull Request Check
Apache License 2.0
89 stars 59 forks source link

The generated annotations have wrong line, file and test name properties #11

Open Vampire opened 4 years ago

Vampire commented 4 years ago

If you might have a look at https://github.com/Vampire/svn2git/actions/runs/167642031 There are three tests failing Your action generates three annotations for that But the annotations are somewhat wrong Line and File are wrong test name is wrong As the report generated by the AutoModality/action-xunit-viewer action seems to be fine, I guess it is not a problem of the XML files being wrong. The XML reports are uploaded as artifacts, so you can have a look at them.

jmisur commented 4 years ago

Hi @Vampire, This action does not support specs (yet). As a lot of reporters are misusing that XML report structure, we always need to adapt for new usages. For example classname in this report is actually a file name, but in Java report it's a real class name. Where do you see correct lines for failed test in the HTML report? All I see is raw logs.

Vampire commented 4 years ago

This action does not support specs (yet).

The point is, I don't care about "specs". Your action says the error was in a spec but this is not nearly correct. The errors were in test/command-line.bats line 30, test/command-line.bats line 40 and test/copy-directories.bats line 27.

I have no idea why your action says the error is on line 1 of file test/libs/bats-core/contrib/rpm/bats.spec which is not even one of my files, just a file of the test harness I'm using. Now that I think about it you maybe take the last dot-separated part which is bats and search for a file with any extension that has name bats which in this case is wrong of course. No idea how the 1 makes it there.

And you also prefix that bats to the test name which of course is wrong in that case.

Where do you see correct lines for failed test in the HTML report? All I see is raw logs.

Yeah, it's probably that raw logs that I have seen. Actually in the raw logs shown by your action the line is contained too of course.

So summarized, actually this issue asks about supporint BATS generated JUnit XML reports. :-)

jmisur commented 4 years ago

As this is very foreign to me, a PR would be appreciated, or at least some form of description about how bats works, some examples of test reports and from where can we extract specific pieces for annotations.