search

vmware-tanzu / sonobuoy

Sonobuoy is a diagnostic tool that makes it easier to understand the state of a Kubernetes cluster by running a set of Kubernetes conformance tests and other plugins in an accessible and non-destructive manner.
https://sonobuoy.io
Apache License 2.0
2.91k stars 344 forks source link

Are you using Sonobuoy? We want to hear from you! #669

Closed johnSchnake closed 1 year ago

johnSchnake commented 5 years ago

We would love to know about your use of Sonobuoy. Share anything Sonobuoy related:

Why?

Tracking usage in OSS is hard, if you leave a comment on this issue we can better understand who our users are and deliver a better experience for YOU!

How you can help

Submit a comment in this issue to include the following information:

Organization/Company: 
Website: 
Country: 
Contact: 
Usage scenario: e.g. We use Sonobuoy for XYZ
Status: Development?/Production?/CI?
Link: [If you are willing, share a link to your awesome script/workflow/plugin so others can learn and benefit]
JTarasovic commented 5 years ago

My company has been using it in an ad-hoc manner to test new K8s releases with our bootstrapping to help verify that everything still runs as expected.

We are looking to automate this in the future so that a new release automatically spins up a cluster, runs a series of tests (including conformance) and outputs the results in a consistent and friendly manner.

I hope that helps and is what you are looking for. Thanks for reaching out to the community!

leodotcloud commented 5 years ago

@johnSchnake I am evaluating this repo to see if we can run the kube-bench plugin.

xdrus commented 5 years ago
ashutoshishere04 commented 5 years ago

how to run only a specific test? how to get that test if you have error message from e2e.log

johnSchnake commented 5 years ago

@ashutoshishere04 You use the --e2e-focus flag (which is the same as setting --plugin-env=e2e.E2E_FOCUS=<value>

See https://sonobuoy.io/docs/v0.16.0/results/ for how to see results in various ways.

Just based on the timing of this question, I would also like to point you towards #910 which found a problem with the k/k testing image which caused it to not save results (and report them to Sonobuoy) when a test error was encountered (on v1.16.0 test image).

Our release of Sonobuoy v0.16.1 today should mitigate that problem though.

ashutoshishere04 commented 5 years ago

@johnSchnake
oh nice my current kubernetes cluster has version=v1.12.8, I'm having 2 failure cases.
failed tests: 2 [sig-network] DNS should provide DNS for services [Conformance] [sig-network] DNS should provide DNS for the cluster [Conformance] apparently their solution is this commit(correct me if i'm confused) https://github.com/kubernetes/kubernetes/pull/43421/commits/3b98c808917bc4947e328d15f47f9f91521c6cc6 I have to use this version1.12.8 only, I've used aks-engine for cluster bootstrapping, is there a way to use solution provided in my current scenario?

Thanks it saved a lot of my time(not running all cases again for one case's result),

johnSchnake commented 5 years ago

@ashutoshishere04 You'll have to look into the exact logs to determine that; I'm not familiar with all the different error modes for these tests. However, the commit you linked to was committed into Kubernetes v1.7.0 so it is a part of the code paths you're using for v1.12.8.

usahai30 commented 5 years ago

Team, I would like to have an understanding on below - As part of regular image(kubernetes) building process, I want to run conformance tests in a pipeline which creates clusters using the image we create everytime. Right now I am using the sonobuoy cmd tool for this. My problem is every-time I have to download this binary in my test pipeline and issue commands through it. Often due to some interrupt my tests breakdown. I am interested to know if somehow I can directly integrate sonobuoy in my test framework in pipeline. By the way - Sonobuoy is an amazing product. I just loved it overall. Thanks

erikwilson commented 4 years ago

k3s is using Sonobuoy for running the conformance tests on every PR, and we try to minimize the runtime as much as possible. To achieve this we create two virtual clusters, one to run the serial portion of the conformance tests, and the other to run the parallel portion of the conformance tests. This allows us to test the full conformance suite in about 20 minutes on multiple architectures (currently amd64 & arm64). We would also like to enable testing for arm but probably need lots of supporting images to be created. We have also disabled syslog support and enabled the e2e go runner.

For reference how we are separating the serial and parallel portion: https://github.com/rancher/k3s/blob/d2756a76c1/scripts/sonobuoy-e2e-tests#L66-L72

And how we are launching sonobuoy itself: https://github.com/rancher/k3s/blob/d2756a76c1/scripts/sonobuoy#L127-L131

Thanks for an awesome utility!

stale[bot] commented 3 years ago

There has not been much activity here. We'll be closing this issue if there are no follow-ups within 15 days.

johnSchnake commented 3 years ago

Stalebot! Just talking about how I thought this issue existed on https://github.com/vmware-tanzu/sonobuoy/issues/1371

We wanted to reopen this so people can record where/how they use Sonobuoy to help share workflows/plugins etc in a very informal way.

negz commented 3 years ago

We use Sonobuoy to implement the Crossplane CNCF conformance suite per https://github.com/crossplane/conformance and https://github.com/cncf/crossplane-conformance.

leodotcloud commented 3 years ago

In Rancher, we use Sonobuoy to invoke kube-bench on all the nodes of a Kubernetes cluster for running CIS scan.

Reference: https://github.com/rancher/security-scan/blob/master/package/run.sh

eddytruyen commented 3 years ago

Hi John,

The following is long overdue but I have published last year a paper about comparing K8-as-a-Service vendors based on analysis of various information sources, one of which are results of the end2end conformance testing suite of Kubernetes.

I've written some code for automating the comparison based on the e2e results (extracted from [conformance-image:1.13, sonobuoy 0.14.3):

It does the following steps:

ubuntu@kubeadm-ubuntu19:~/githubrepos/MigratingKubernetes/validation$ ./run.sh
1. Extracting and classifying sonobuoy's e2e-tests => junit.xml files and e2e-test logs per K8S vendor aggregated in directory run-results...
2a. Processing all tests: classifying succeeded and failed tests => For each K8s vendor, 2 csv files are generated in directory run-results: one with succeeded tests and one with failed tests. This processing may take several minutes...
2b. Processing all tests: counting number of succceeded and failed tests and generating summary => run-results/Summary.csv....
Run completed successfully!
To delete all results of this run, execute './cleanup.sh'

As you can see, my code does similar things as sonobuoy results. It is possible to aggregate results from multiple tar balls from multiple vendors and the main output is a Summary.csv file. Each entry in the csv corresponds with a documented k8s feature, a specific set of e2e test regular expressions that relate to that k8s feature, and per vendor the number of succeeded and failed tests.

For each vendor there are two other files with the same structure as Summary.csv. "vendor_fail.csv" en "vendor_success.csv". Each entry contains the snippets of the junit file that correspond with the selected tests.

I am now starting a new project based on this work and I did a very small test of my code for output of [Kubernetes-conformance image19.4, sonobuoy 0.52.3]. It still seems to work. However it contains some hacky grep commands some of which maybe replaced to invocations of sonobuoy results | jq

Maybe my code is usable in some sense.

Do you know of related projects?

Below you find the references

Don't hesitate to contact me in case you have questions,

Best regards,

Eddy

johnSchnake commented 3 years ago

@eddytruyen I think the paper link should be https://ieeexplore.ieee.org/document/9298825

Thanks so much for your submission, I'm definitely going to take some time to read through. Sounds great though :)

Update: I just wanted to comment a few things for visibility for people looking at the paper/our comments later.

I'm so thrilled that we were of use in your analysis. Thank you for sharing this with us all!

matthias50 commented 3 years ago

I've been looking at sonobuoy to replace our home grown tools for running k8s e2e tests. So far it looks like a huge improvement. My favorite feature so far the ability to easily re-run just the e2e tests which failed previously. One usability nit I can see is that if you are using air gapped clusters (or gatekeeper), you have to specify command line flags to point to various config which will map the location to pull the sonobuoy image as well as the images used by the e2e tests themselves.

While this will obviously work, it would be nice if sonobuoy would also look for config in some default location like say ~/.sonobuoy. That way one could do a one time config setup and then just use the "normal" CLI without having pass the extra options. Thanks!

johnSchnake commented 3 years ago

it would be nice if sonobuoy would also look for config in some default location like say ~/.sonobuoy

Thanks for the feedback! We just introduced that location as the home for plugins so it does open the gates for ideas/use cases like this. 👍 @matthias50

stale[bot] commented 1 year ago

There has not been much activity here. We'll be closing this issue if there are no follow-ups within 15 days.