Issues with Venom v1.1.0-beta.5 JSON report

[Thiht]

Tested on these test suites: https://github.com/Thiht/vscode-venom/tree/master/sample

With the following command: /tmp/venom-v1.1.0-beta.5 run --format=json --output-dir=/tmp/out '**/*.venom.yml'

---

Venom creates 1 report per test suite:

$ ls -1
test_results.error.venom.yml.json
test_results.failure.venom.yml.json
test_results.success.venom.yml.json

That's a bit annoying to parse, is it possible to get back to a unique file?

---

Also there's a bug, because I have 2 test suites named success.venom.yml (sample/success.venom.yml and sample/nested/success.venom.yml) but it generates only one report because of the naming scheme test_results.<filename>.json (can be fixed if you go back to a unique file :p )

---

Last bug I see is that in the line numbering is broken in the error value message. Before:

"failures": [
  {
    "value": "\u001b[33mTestcase \"Sample test case #2\", step #0: Assertion \"result.systemout ShouldEqual \\\"Good Bye\\\"\" failed. expected: Good Bye  got: Hello (failure.venom.yml:13)\u001b[0m",
    "type": "",
    "message": ""
  }
],

(note the failure.venom.yml:13 at the end of the message)

After:

"errors": [
  {
    "value": "Testcase \"Sample test case #2\", step #0-0: Assertion \"result.systemout ShouldEqual \\\"Good Bye\\\"\" failed. expected: Good Bye  got: Hello (:0)"
  }
],

If not in the error message, can you include the line number in the assertion itself? If you use gopkg.in/yaml.v3 it's possible to get the line number at parse time with a custom unmarshaller:

func (a *Assertion) UnmarshalYAML(value *yaml.Node) error {
    if err := value.Decode(&a.Raw); err != nil {
        return err
    }
    a.LineNumber = value.Line

        // ...

see: https://github.com/go-yaml/yaml/blob/v3.0.1/yaml.go#L372

[yesnault]

Thank you for the report.

We keep one output file per testsuite. This avoid to override results when venom is executed in bash loop. It's probably possible to re-assemble json file with another tool as jq. We'll probably add others commands in 1.2 as venom tool merge *.json and venom tools html-report *.json to make it easier.

The same names are now correctly handled.

The lineNumber detection is fixed, I'm not sure if it's ok in every case (after interpolation, assertion in user executor) but it works as before now.

after fix, output is now:

 venom run --format=json --output-dir=/tmp/out '**/*.venom.yml'
      [trac] writing /tmp/out/venom.log
 • Another Venom test suite (success.venom.yml)
    • Sample-test-case-1 PASS
    • Sample-test-case-2 PASS
 • Sample Venom test suite (failure.venom.yml)
    • Sample-test-case-1 PASS
    • Sample-test-case-2 FAIL
Testcase "Sample test case #2", step #0-0: Assertion "result.systemout ShouldEqual \"Good Bye\"" failed. expected: Good Bye  got: Hello (failure.venom.yml:13)
 • Sample Venom test suite (error.venom.yml)
    • Sample-test-case-1 FAIL
error decoding assertions: 1 error(s) decoding:

* 'Assertions': source data must be an array or slice, got map
 • Another Venom test suite (nested/success.venom.yml)
    • Sample-test-case-1 PASS
    • Sample-test-case-2 PASS
Writing file /tmp/out/test_results_success.venom.json
Writing file /tmp/out/test_results_failure.venom.json
Writing file /tmp/out/test_results_error.venom.json
Writing file /tmp/out/test_results_nested_success.venom.json
final status: FAIL