Open jonwinton opened 2 months ago
Oh, looks like the JSON case is empty
Attempting to fix in this PR: https://github.com/kyverno/chainsaw/pull/1538
Happy to make any necessary changes
I'm not sure if junit supports json 🤔 Trying to confirm that
Ah that would be a fun gotcha 😅
For context, I really like the idea of a json output because I'm trying to build some continuous automated testing for some clusters and being able to parse the test results would be super helpful. If json should not be a supported output I'm also happy to just remove that option entirely, but I would say that I love the idea of having the report in json format even if the json output isn't junit compatible.
Yeah, i agree, i'm not a test report format expert but it looks like junit only supports xml officially.
I'd like to find a json equivalent if it exists.
@eddycharly I've done some searching around and have found some javascript projects that a junit style json conversion, like this one. Is this what you're imagining for the feature? I'd be happy to implement something like this, if so.
That said, I'm not sure if this would be in "bug" scope. Would we be able to consider the lack of JSON output a bug and go forward being able to output the json file in the current format of the Report
struct and then make it a breaking change to update the output format to junit style as a breaking change?
I actually think I've uncovered an XML report bug as well with the file output not rendering failures properly. For context, I'm working on a test harness to execute chainsaw
in Kuberhealthy (which is working really nicely and I'm loving it), and I'm now working on improving the error surfacing. I'd love to invest a little time into the reporting if you're looking for someone to do any work in that area.
Ok, sorry for all the noise, but I took a quick stab at fixing a couple of things I was running into for my own purposes: https://github.com/kyverno/chainsaw/pull/1539
On my side project I'm just using yq
to handle my XML --> JSON conversion quickly but I'm still interested in finding the most ideal reporting fix so I can just use native output. Let me know if there's anything I can help collaborate on! I also just joined the chainsaw channel in the Kubernete slack and am happy to talk more there if it's better
I've been thinking about a good fix too. One option would be to use the raw output from the golang testing package and defer to tools that work with the native output. While i like this solution i feel like it will lack test details. I need to think about it more :(
I think the best approach is to create an internal report and use conversion to specific formats.
@eddycharly I'm down to work on this but won't have time until next week at the earliest.
chainsaw version Version
v0.2.4
Description
I'm trying to generate a JSON report to then parse for errors but the report is never being generated. I'm following the docs here using the getting started test. It works when
--report-format
is set toXML
, but notJSON
.I checked
v0.1.9
and theJSON
output worked, butv0.2.0
did not. I'm wondering if it's related to the changes from this feature: https://github.com/kyverno/chainsaw/issues/1099.Steps to reproduce
chainsaw test --report-name chainsaw-report --report-format JSON --report-path "$PWD"
. No report should be generated.chainsaw test --report-name chainsaw-report --report-format XML --report-path "$PWD"
. A report should be generatedExpected behavior
When
--report-format
is set toJSON
a report will be generated.Screenshots
No response
Logs