Add Sonobuoy progress and structured results

[johnSchnake]

Utilizes the Sonobuoy plugin-helper to show progress as tests run and to report results in a structured format so that we can get result counts more easily/programatically.

The way the output looks then is:

sonobuoy run -p ./sonobuoy-plugin.yaml --wait --image-pull-policy=Always --level=trace
TRAC[0000] Invoked command run with args [] and flags [image-pull-policy=Always level=trace plugin=k8s-service-validator-sonobuoy-plugin wait=1440]
TRAC[0000] Resolved 'auto' version to v1.21.2 from GitVersion v1.21.2+vmware.1-360497810732255795
INFO[0000] create request issued                         name=sonobuoy namespace= resource=namespaces
INFO[0000] create request issued                         name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] create request issued                         name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] create request issued                         name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
INFO[0000] create request issued                         name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] create request issued                         name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0000] create request issued                         name=sonobuoy namespace=sonobuoy resource=pods
INFO[0000] create request issued                         name=sonobuoy-aggregator namespace=sonobuoy resource=services
13:26:54                                   PLUGIN     NODE    STATUS   RESULT   PROGRESS
13:26:54    k8s-service-validator-sonobuoy-plugin   global   running
13:26:54
13:26:54 Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.
...
...
...
...
13:28:34    k8s-service-validator-sonobuoy-plugin   global   running            1/5 (1 failures)
13:28:54    k8s-service-validator-sonobuoy-plugin   global   running            2/5 (1 failures)
...
13:29:34    k8s-service-validator-sonobuoy-plugin   global   complete            5/5 (2 failures)
13:29:34 Sonobuoy plugins have completed. Preparing results for download.
13:29:54    k8s-service-validator-sonobuoy-plugin   global   complete   failed   5/5 (2 failures)
13:29:54 Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
➜  k8s-service-validator git:(sonobuoyHelpers) ✗ sonobuoy retrieve -f out.tar.gz
out.tar.gz
➜  k8s-service-validator git:(sonobuoyHelpers) ✗ sonobuoy results out.tar.gz
Plugin: k8s-service-validator-sonobuoy-plugin
Status: failed
Total: 5
Passed: 3
Failed: 2
Skipped: 0

Failed tests:
TestBasicService
TestLabels

So we get:

• progress updates while running
• structured output in sonobuoy_results.yaml
• integration with sonobuoy results <tarball> (due to the above)

Fixes #113 Fixes #102

[johnSchnake]

I ran it locally with this patch for the e2e-framework https://github.com/kubernetes-sigs/e2e-framework/pull/102 and confirmed we saw 12 features complete and I got pass/fail info on them.

We'd just have to wait for it to merge

[johnSchnake]

I'd still like to pair with someone about the failures in CI. I still need to walk through it a bit more, but I've seen some code paths that make it look like its not building/pushing the new plugin image into the cluster to test. I'm guessing I'm wrong, but I haven't spent much time looking at the CI for this project yet.

[johnSchnake]

Oooh I think I understand why its complaining about tarring things up. Now that the sonobuoy logic is in the code, it is still trying to run that logic even though the tests themselves are just running via go test. Before that logic was in the plugin, which actually doesn't get run in CI. So I wasn't crazy, but neither is the bug. Locally I was testing it as the plugin, not as go test

[johnSchnake]

I modified the go.mod to just use replace ... for my own fork for my own testing. Just wanted to push here to ensure it doesn't do the erroring related to tarring up the results directory.

I patched the pluginHelper to work in this case (running not in a cluster) and confirmed locally that I didnt get that error.

[yzaccc]

I ran it locally with this patch for the e2e-framework kubernetes-sigs/e2e-framework#102 and confirmed we saw 12 features complete and I got pass/fail info on them.

We'd just have to wait for it to merge

That's great!

[johnSchnake]

e2e-framework PR merged; now we can worry more about this passing CI and merging