Improve testing with stats and errors per test section

[jjmaestro]

Add some basic test stats to the test runner output for both SimpleReport and JUnitReport:

• number of tests that pass/fail
• % of success
• number of assertions within each test that pass / fail

[bioball]

Thanks for the PR! FYI: I probably won't get around to reviewing this until after our 0.26 release is out in early June.

[jjmaestro]

@bioball no worries! I actually submitted the PR too early :S I'm re-doing it all because (1) I didn't realize I had a bunch of tests failing and (2) the failures and other stuff made me re-think it all.

I'm currently thinking about how to best approach things... I might close the PR and re-open a new one once I have something that 100% works. E.g. I thought I was going to be able to change some of the XML format, thinking it was a bit freeform and I'm now learning about the different XML schemas for unit testing 😅 Still not sure which one Pkl is using and/or if I can change it to add more of the information I'd like to have, etc.

Anyway, TL;DR no rush :) Thanks!!

[jjmaestro]

OK, I think I finally got it done :) I've refactored a bunch of stuff and fixed my original commit, here's the summary:

• TestRunner: separated the running of each "section" of a test into its own try-catch so that we can evaluate each section separately even if another section failed (e.g. evaluate all examples even if one test in "facts" throws an exception). I've added tests that exercise this "new feature".

• TestResults: refactored it a bit so that TestResults now has the three sections and the old TestResults becomes TestSectionResults. Also, TestSectionResults now has only one error, since that's what we get when there's an exception evaluating the Pkl code.

• SimpleReport:

  • Separated reporting of each test section so that it's easy to see what facts or what examples fail.
  • Added a "stats line" to the module and section levels that reports how many tests / assertions pass/fail. Note that, when reporting errors, there's no stats for the section since the error is all the info we get. Similarly, when examples fail, there's no stats because each example is converted into its own test so there are no failed assertions to report, it's all-or-nothing.
  • For the examples converted to tests, when multiple examples share the same name I've also added a counter ("# 1" and so on) so that it's easier to identify which one is failing.

• JUnitReport: fixed the reporting of failures as it wasn't consistent with "tests": the tests were the number of tests that we run but "failures" were the number of assertions that failed.

Here's some outputs that show these new features:

• Some facts pass, some fail and some examples pass, with stats:

  module test1 ❌ 66.7% pass [2 passed, 1 failed] (file:///private/tmp/wut/test1.pkl, line 1)
  facts ❌ 0.0% pass [0 passed, 1 failed]
  foo ❌ 50.0% pass [1 passed, 1 failed]
    10 == 11 ❌ (file:///private/tmp/wut/test1.pkl, line 6)
  examples ✅ 100.0% pass [2 passed]
  user 0✍️
  user 1✍️

• Facts throws an error but now, we can still evaluate examples passing and failing:

  module test ❌ 33.3% pass [1 passed, 2 failed] (file:///tmp/test.pkl, line 1)
  facts ❌
  Error:
  –– Pkl Error ––
  exception1
  
  9 | throw("exception1")
      ^^^^^^^^^^^^^^^^^^^
  at test#facts["error1"][#1] (file:///tmp/test.pkl, line 9)
  
  3 | facts {
      ^^^^^^^
  at test#facts (file:///tmp/test.pkl, line 3)
  examples ❌ 50.0% pass [1 passed, 1 failed]
  user 0 ✅
  user 1 #0 ❌
    (file:///tmp/test.pkl, line 25)
    Expected: (file:///tmp/test.pkl-expected.pcf, line 9)
    new {
      name = "Pigeon"
      age = 41
    }
    Actual: (file:///tmp/test.pkl-actual.pcf, line 9)
    new {
      name = "Pigeon"
      age = 40
    }

[jjmaestro]

Ah, so I don't forget... I've been trying to find out if there's an XSD to validate the XML produced by JUnitReport and I eventually found Jenkins' xUnit plugin that lists a bunch of supported tools. In that list, I saw it mentions JUnit support for two schemas: Ant junit and Maven Surefire.

I checked both and Maven Surefire is the closest to the XML JUnitReport produces... however, it does not validate against the XSD (many elements are missing required attributes).

Is there a tool that loads the XML report that could serve as a validator of sorts? :-?

[bioball]

I've tried many times to look for a schema, but as far as I can tell, there's no real schema for JUnit reports. And, the many tools that create JUnit reports all differ in minor ways.

Tools like Jenkins that accept JUnit-style reports tend to just make a best-effort attempt to parse them, so, we just do the best we can to be conformant to what's out there.

[jjmaestro]

Arrrg CI failed because of linting! I could have sworn I did a final gw spotlessApply before commiting :( Anyway, I've just re-committed it, hopefully it'll all be fine now!

[jjmaestro]

OK, I've just pushed the resolved merge conflict... it passes all the same tests. @bioball is there any way to ensure the commit is not blocked in CI? I've seen it blocked ("on hold") for ~a week before :-/ Is there any reason why that happens? :-?

Thanks!!

[jjmaestro]

Tests failed for gradle-check-jdk17-windows!? 😮 Any help to repro these would be very welcomed, I've read the CI output and I'm not too sure what's failing / what's going on... especially since the tests seem to be the same tests that pass other CI runs and pass on my laptop! :-?

EDIT: ah, I think the error is the following but I'm not too sure: java.nio.file.NoSuchFileException: C:\Users\circleci\AppData\Local\Temp\junit15390432245720982242\build\test.xml

I usually double-check the reports like pkl-gradle/build/reports/tests/test/index.html but I don't know if it's possible to access those via the CircleCI interface.