Post-processing: Add new messages

[johnSchnake]

Should be able to add new context messages to individual tests

[johnSchnake]

Q: Is this just for the e2e results? The manual sonobuoy format? The gojson?

There won't be a single method of doing this for every format unless we introduce a separate piece of logic into the aggregator to do the merging between different types (e.g. we write the modifications in the yaml format but the aggregator looks for this known file and merges it after processing the plugin results)

Currently the change that went into Sonobuoy to facilitate the post-processing was just a delay window so that other containers can "swoop in" and remove the done file in order to do what they want with the results.

In that context, to add new messages means to:

• wait for the donefile (poll more frequently than the worker will wait)
• have preconfigured data (the data to inject, eg KB articles or something, potentially at URLs?)
• grab results (specified in the done file)
• merge results file with the preconfigured data

This would require us to have an image that:

• does the polling/waiting for the done file (ideally configurable; will probably make chaining these easier if you can specify all in/out files)
• loads preconfigured data from a URL or other source (it won't be baked into the generic image)
• accepts the expected name of the file to merge results into (e.g. the original plugin results file). We may be able to settle for the type of file if we anticipate only a single file of that type present. That way they just say 'xml' or 'junit' instead of having to know its called 'e2e_results.xml' (I'm not even certain if thats what it is named)

It seems reasonable that this one image should be able to handle different types of data, so perhaps we can have a single image that will handle junit, gojson, manual sonobuoy format, etc.

For the first iteration, the format for the e2e tests is a good test.

[johnSchnake]

Another question is whether or not we must only add new messages when matching the testname exactly, or are we going to allow things like comparing by regexp.

Even if the preconfigured data has to be of a specific format, it would make sense for the key-matching to be configurable (e.g. regexp or exact).

Just brainstorming here, but the question I still have is whether or not providing a results-like-file with a few options and merging them with the real results is the best or only way to do this.

We could also allow the data provided by the user/URL to be a series of JSON patch type elements. Then they could add/delete/modify them as necessary. It seems like it would be harder to make a series of json patches though than simply "copying" the format of the desired output and saying to merge. I'm also not certain json patch can do things like matching keys based on value (e.g. update the output for the test with this name...) rather than having to use indices (tests[0])

[johnSchnake]

For the e2e tests, cis-benchmarks, and other plugins which produce junit (xml) we could rely on xslt and allow users to go wild. That has the benefit of not being so home-spun and being generic enough to handle anything. I haven't used that in a looong time though so I'm not sure if there are limitations we need to be concerned about.

[johnSchnake]

Reading more about peoples feelings of xslt I may not want to go that route; lots of people don't seem to find it easy to work with. Anyone could write a very generic xslt transform though, but we'd rather focus our effort on the more typical use cases of dealing with test results and:

• adding extra output
• removing tests (e.g. skipped tests or ones you dont think are relevant but can't easily skip)
• splitting up output into files for easier consumption (skips/pass/fail each in their own files would be helpful)
• modifying output or results based on some external data

I think this can be pretty easily accomplished if we just allow the users to provide their own data set with some extra metadata fields. For instance, if tests look like this:

<testcase name="foo"><output>abc</output></testcase>

then maybe modifications could look this this:

<testcase name="foo" sonobuoy-matcher="exact" sonobuoy-action="append"><output>def</output></testcase>

Then we could just define a handful of matcher (exact/regexp) and actions (append, add, remove, replace, etc). A few details to work out but then the way its processed is pretty straight-forward.

I'm going to flush this out in a DD but I think I'm leaning this way as it is specific to the test case uses and I think simple to use.

[johnSchnake]

Through the design process we decided to go with utilizing ytt to do the bulk of the work so most of these questions are trivial/moot.

Will review the notes to see if I can close this completely. There were some ideas which would have to be outside of the ytt logic.