[Bug] Error creating report in JSON format in v0.2.x

jonwinton commented 2 months ago

chainsaw version Version

v0.2.4

Description

I'm trying to generate a JSON report to then parse for errors but the report is never being generated. I'm following the docs here using the getting started test. It works when --report-format is set to XML, but not JSON.

I checked v0.1.9 and the JSON output worked, but v0.2.0 did not. I'm wondering if it's related to the changes from this feature: https://github.com/kyverno/chainsaw/issues/1099.

Steps to reproduce

Setup a directory with the getting started test
Run chainsaw test --report-name chainsaw-report --report-format JSON --report-path "$PWD". No report should be generated.
Run chainsaw test --report-name chainsaw-report --report-format XML --report-path "$PWD". A report should be generated

Expected behavior

When --report-format is set to JSON a report will be generated.

Screenshots

No response

Logs

❯ chainsaw test --report-name chainsaw-report --report-format JSON --report-path "$PWD" 
Version: 0.2.4
Loading default configuration...
- Using test file: chainsaw-test
- TestDirs [.]
- SkipDelete false
- FailFast false
- ReportFormat 'JSON'
- ReportName 'chainsaw-report'
- ReportPath '/Users/jwinton/Development/example-test'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 5s
- AssertTimeout 30s
- CleanupTimeout 30s
- DeleteTimeout 15s
- ErrorTimeout 30s
- ExecTimeout 5s
- DeletionPropagationPolicy Background
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- quick-start (.)
Loading values...
Running tests...
=== RUN   chainsaw
=== PAUSE chainsaw
=== CONT  chainsaw
=== RUN   chainsaw/quick-start
=== PAUSE chainsaw/quick-start
=== CONT  chainsaw/quick-start
    | 08:23:50 | quick-start | @setup   | CREATE    | OK    | v1/Namespace @ chainsaw-helpful-hagfish
    | 08:23:50 | quick-start | step-1   | TRY       | RUN   |
    | 08:23:50 | quick-start | step-1   | APPLY     | RUN   | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | CREATE    | OK    | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | APPLY     | DONE  | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | ASSERT    | RUN   | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | ASSERT    | DONE  | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | TRY       | DONE  |
    | 08:23:50 | quick-start | step-1   | CLEANUP   | RUN   |
    | 08:23:50 | quick-start | step-1   | DELETE    | RUN   | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | DELETE    | OK    | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | DELETE    | DONE  | v1/ConfigMap @ chainsaw-helpful-hagfish/chainsaw-quick-start
    | 08:23:50 | quick-start | step-1   | CLEANUP   | DONE  |
    | 08:23:50 | quick-start | @cleanup | DELETE    | RUN   | v1/Namespace @ chainsaw-helpful-hagfish
    | 08:23:50 | quick-start | @cleanup | DELETE    | OK    | v1/Namespace @ chainsaw-helpful-hagfish
    | 08:23:56 | quick-start | @cleanup | DELETE    | DONE  | v1/Namespace @ chainsaw-helpful-hagfish
--- PASS: chainsaw (0.00s)
    --- PASS: chainsaw/quick-start (5.36s)
PASS
Tests Summary...
- Passed  tests 1
- Failed  tests 0
- Skipped tests 0
Done.



### Slack discussion

_No response_

### Troubleshooting

- [X] I have searched other issues in this repository and mine is not recorded.

jonwinton commented 2 months ago

Oh, looks like the JSON case is empty

jonwinton commented 2 months ago

Attempting to fix in this PR: https://github.com/kyverno/chainsaw/pull/1538

Happy to make any necessary changes

eddycharly commented 2 months ago

I'm not sure if junit supports json 🤔 Trying to confirm that

jonwinton commented 2 months ago

Ah that would be a fun gotcha 😅

For context, I really like the idea of a json output because I'm trying to build some continuous automated testing for some clusters and being able to parse the test results would be super helpful. If json should not be a supported output I'm also happy to just remove that option entirely, but I would say that I love the idea of having the report in json format even if the json output isn't junit compatible.

eddycharly commented 2 months ago

Yeah, i agree, i'm not a test report format expert but it looks like junit only supports xml officially.

I'd like to find a json equivalent if it exists.

jonwinton commented 2 months ago

@eddycharly I've done some searching around and have found some javascript projects that a junit style json conversion, like this one. Is this what you're imagining for the feature? I'd be happy to implement something like this, if so.

That said, I'm not sure if this would be in "bug" scope. Would we be able to consider the lack of JSON output a bug and go forward being able to output the json file in the current format of the Report struct and then make it a breaking change to update the output format to junit style as a breaking change?

jonwinton commented 2 months ago

I actually think I've uncovered an XML report bug as well with the file output not rendering failures properly. For context, I'm working on a test harness to execute chainsaw in Kuberhealthy (which is working really nicely and I'm loving it), and I'm now working on improving the error surfacing. I'd love to invest a little time into the reporting if you're looking for someone to do any work in that area.

jonwinton commented 2 months ago

Ok, sorry for all the noise, but I took a quick stab at fixing a couple of things I was running into for my own purposes: https://github.com/kyverno/chainsaw/pull/1539

On my side project I'm just using yq to handle my XML --> JSON conversion quickly but I'm still interested in finding the most ideal reporting fix so I can just use native output. Let me know if there's anything I can help collaborate on! I also just joined the chainsaw channel in the Kubernete slack and am happy to talk more there if it's better

eddycharly commented 2 months ago

I've been thinking about a good fix too. One option would be to use the raw output from the golang testing package and defer to tools that work with the native output. While i like this solution i feel like it will lack test details. I need to think about it more :(

eddycharly commented 2 months ago

I think the best approach is to create an internal report and use conversion to specific formats.

jonwinton commented 2 months ago

@eddycharly I'm down to work on this but won't have time until next week at the earliest.

kyverno / chainsaw