soenkeliebau
commented
2 years ago There was a discussion about this on Slack for those who are interested: https://kubernetes.slack.com/archives/CG3HTFCMV/p1649681144207229
Hi all,
we are using Kuttl for our tests and it has really made testing live much easier. However we have a need to test many scenarios which at the moment require a lot of copy and pasting of test cases.
We'd like to extend Kuttl to allow for parameters in test cases, but wanted to check if the community would be willing to consider an extension like this.I have created https://github.com/kudobuilder/kuttl/issues/350 for this purpose, but not received much feedback so far, so I wanted to ask whether that is the right place for something like this, or if I should rather ask somewhere else, create a KEP, ...
GitHub
What would you like to be added: Problem Currently Kuttl does not allow creating something like a test matrix which would allow testing any combination of a number of parameters. For example testin... (93 kB)
https://github.com/kudobuilder/kuttl/issues/350
1
Sönke Liebau Apr 11th at 2:45 PM
Hi all,
we are using Kuttl for our tests and it has really made testing live much easier. However we have a need to test many scenarios which at the moment require a lot of copy and pasting of test cases.
We'd like to extend Kuttl to allow for parameters in test cases, but wanted to check if the community would be willing to consider an extension like this.I have created https://github.com/kudobuilder/kuttl/issues/350 for this purpose, but not received much feedback so far, so I wanted to ask whether that is the right place for something like this, or if I should rather ask somewhere else, create a KEP, ...
GitHub
What would you like to be added: Problem Currently Kuttl does not allow creating something like a test matrix which would allow testing any combination of a number of parameters. For example testin... (93 kB)
https://github.com/kudobuilder/kuttl/issues/350
1
42 replies
I would like to see something in this direction added to kuttl. But it is a «minefield» of decisions to make, and the API/DSL isn’t obvious.
Personally I prefer patching over templating, since I find it more declarative. Could kustomize support in kuttl be an option?
1
I think we mostly went down the templatng route because that was something that most of the team was familiar with, so it felt like a natural solution.
One issue I see with patching for our specific usecase is, that we use external scripts to validate test results above and beyond what Kuttl provides. So basically Kuttl tests that k8s says "yep, all running" and then we start a script that goes and queries an api to see that app state is green and deployed in the correct version (or similar).
Patching would be limited to yaml files unless I misunderstand, whereas templating would allow a more generic approach.
I understand, but templating can get really messy. 
Not trying to defend templating in general 
I have seen (and probably also created) some horrible templates
In the end it all starts with yaml in this case, even the scripts are called by Kuttl, so the call itself is defined in yaml, which would be subject to patching..
And using something like kustomize will take away a lot of the discussions related to the API/DSL. WDYT @kensipe?
How would you envision the orchestration around this, would you want to enable users to add a list of patch files and then run a test scenario per patch file?
I would suggest that any test case in kuttl could be considered a kustomization of another one.
That sounds useful too, but I think wouldn't address our specific use case. This sounds more like test case inheritance?
Call it what you like, but kustomize is not just about inheritance. You can define components, replacements etc. And there are quite a few kustomize patterns, like diamonds.
Full disclosure: I am far removed from being an expert on Kustomize, so will probably need to do some reading
What I don't like about your current apporach, if I have understood it correctly, is the custom DSL you need to learn. It might work well in your use case, but the hard thing is to make something generic useful for more users...
I am absolutely not trying to promote our exact approach here, please don't get me wrong, that is hacky, ugly and was just born out of a need to have something working right now.
This is the most important thing to understand about kustomize: https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/
SIG CLI
Eschewed Features
Not having variables seems to be the most frustrating issue for people, but also a good thing!
I guess we have been taught to think like that for decades
I have been over my head with issues in templating engines like Ansible/Jinja and Helm, so I prefer kustomize now. Even if some things are hard to achieve.
I am probably right on the other side of the fence, not all sold on Ansible, but its the known evil thats there and just works most of the time ..
But regarding our solution (again, not promoting it, just using it as basis for discussion): which custom DSL are you concerned about?
The test definitions part, or the templating part ?
Without having looked to much on the details, it seems like the "dimension" part isn't flexible enough. Over time I think it could get really messy. For comparison, have a look at the corresponding GitLab CI/CD feature: https://docs.gitlab.com/ee/ci/yaml/#parallelmatrix. We are using it, that's why I am aware of it. (edited)
docs.gitlab.com
Keyword reference for the .gitlab-ci.yml file | GitLab
Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.
That looks pretty much like what we are trying to achieve, yes.
It must also define a very loose structure, which can make it very hard to validate. Ending up in hours of bug-tracking if something is wrong. (edited)
Loose because otherwise it'll stop working at some point as use cases evolve?
Loose in the schema sense.
I'm with you most of the way, this is trying to address a very generic use case, so would by its very nature tough to validate properly.
But as I wrote, I would like to see something in this direction. Either using templating or patching (preferred IMO). 
Maybe @iblancasa has something to add?
I think that is the fundamental thing we need to agree on, everything else can be called an implementation detail for now
@Sönke Liebau thanks for the ping... thanks for the issue submission. I will take a look shortly
no rush, we have something that works for us, the issue is more the long con
@erikgb regarding kustomize... we had discussed at some point in the past. If I remember, we were leaning towards starlark... some kind of tooling would be useful.
1
@kensipe It would be nice to review the arguments used in the discussion you are referring to. Any pointers?
oh... it's lost in history most likely. it was discussions that happened > 2 years ago. I will look
Starlark would be more in the templating direction, right? If i remember correctly, it is a pythonic Go tool. If people are more comfortable with templating, I am ok with that. Maybe we could support some kind of "templating engine interface", that will include the DSL to configure kuttl. And the option of pluggable templating engines, defaulting to Starlark. That will allow people to use the templating tool they are used to. All are very complex and hard to master IMO.
1
In my case there is no strong preference for templating over patching to be honest, however I currently don't see how pure patching would fit our use case. Anything I can think of would require generating the patches with templating or something similar.
That being said, our use case is just that, our use-case. If Kuttl ends up implementing something that doesn't solve our case because we are too far off the beaten path, that is totally all right.
@Sönke Liebau ok... caught up on issues and context. The concept is familiar to me as we had the same kind of matrix. I'm a little confused why java would be a dimension of the matrix as the runtime shouldn't impact the functionality which is what kuttl would be testing. I could see changes in container support / resource differences but that doesn't seem to align with kuttl testing. (mention some of this in case there is value in a deeper discussion). Sticking to the need for a matrix (regardless of reason)... I imagine there is something that would need to templated to support this (which could be why the slack thread mentions kustomize. (I haven't read the entire slack thread yet... getting to that next). This use case could be an interesting driver for templating capabilities... perhaps some refinement on the details. Are you thinking that an early test step is installation of 1 element of the dimension? Assuming that there is a delete step with the same value.
To answer the last part of the original post... yes... a KEP with details would be a great place to start
Regarding the java part of your question, this is not an actual requirement we have at the moment, I just replaced "spark" and "hadoop" from our matrix with something more generally well known for the issue.
1
That being said, in our test suite there actually may come a time when we want to run the same test case with containers offering different runtimes.
We are aware that this goes beyond the intended purview of Kuttl and moves us into functional tests to a certain extent.
However I'd argue that there is value in having Kuttl be the common denominator that drives tests for both integration tests and functional tests.
Are you thinking that an early test step is installation of 1 element of the dimension? Assuming that there is a delete step with the same value.
The way we currently do it is that the test case itself really is just the template and multiplied into test scenarios using the dimensions.An example is here: https://github.com/stackabletech/spark-k8s-operator/tree/068aac797b9354a9de3c217af269ed010d0144b0/tests/templates/kuttlOur pre-Kuttl ansible playbook takes this and the dimensions file and "renders" the template test case into _work directory which then has roughly this structure (not sure how this will render in Slack..):├── kuttl-test.yaml
├── tests
│ ├── spark-pi-private-s3
│ │ ├── spark-3.2.1_hadoop-3.1.1_stackable-0.4.0
│ │ │ ├── 00-assert.yaml
│ │ │ ├── 00-s3-secret.yaml
│ │ │ ├── 00-s3-upload-container.yaml
│ │ │ ├── 00-setup-minio.yaml
│ │ │ ├── 01-prepare-bucket.yaml
│ │ │ ├── 02-assert.yaml
│ │ │ ├── 02-deps-volume.yaml
│ │ │ ├── 03-assert.yaml
│ │ │ ├── 03-list-jars.yaml
│ │ │ ├── 10-assert.yaml
│ │ │ ├── 10-deploy-spark-app.yaml
│ │ │ └── spark-examples_2.12-3.2.1.jar
│ │ ├── spark-3.2.1_hadoop-3.2.0_stackable-0.4.0
│ │ │ ├── ...
│ │ ├── spark-3.2.2_hadoop-3.1.1_stackable-0.4.0
│ │ │ ├── ...
│ │ └── spark-3.2.2_hadoop-3.2.0_stackable-0.4.0
│ │ ├── ...
│ └── spark-pi-public-s3
│ ├── spark-3.2.1_hadoop-3.1.1_stackable-0.4.0
│ │ ├── ...
│ ├── spark-3.2.1_hadoop-3.2.0_stackable-0.4.0
│ │ ├── ...
│ ├── spark-3.2.2_hadoop-3.1.1_stackable-0.4.0
│ │ ├── ...
│ └── spark-3.2.2_hadoop-3.2.0_stackable-0.4.0
│ ├── ...And that directory then is what we run Kuttl on
GitHub
Operator for Apache Spark-on-Kubernetes for Stackable Data Platform - spark-k8s-operator/tests/templates/kuttl at 068aac797b9354a9de3c217af269ed010d0144b0 · stackabletech/spark-k8s-operator (67 kB)
GitHub
spark-k8s-operator/test-definition.yaml at main · stackabletech/spark-k8s-operator
Operator for Apache Spark-on-Kubernetes for Stackable Data Platform - spark-k8s-operator/test-definition.yaml at main · stackabletech/spark-k8s-operator (67 kB)
https://github.com/stackabletech/spark-k8s-operator/blob/main/tests/test-definition.yaml
this is helpful.. I'll dig in to understand more.. thanks
Would it be ok for the two of you if I attached this thread to the issue I created?
1
What would you like to be added:
Problem Currently Kuttl does not allow creating something like a test matrix which would allow testing any combination of a number of parameters. For example testing an operator that rolls out a product would require copy and pasting test cases for every version that is under testing. This becomes even worse when the test is dependent on multiple dimensions, for example product version and java version.
Consider as an example the following dimensions:
Product Version:
Java Version:
This would require copy and pasting the same test case 12 times and adjusting the values manually in the test files, which is error prone, laborious and hard to maintain.
Proposal
We would like to be able to define test dimensions for Kuttl tests which can then be used to parameterize tests and effectively create a test matrix at runtime.
We have implemented external tooling around Kuttl at the moment to provide this functionality based on a test dimensions file that has the following structure:
This allows defining an arbitrary number of test dimensions along with values, as well as test cases that use these dimensions and which dimensions they actually use. Our tooling then creates the needed list of test cases and runs a templating engine over the input kuttl files to generate the actual test files into those test case directories.
For the above example we'd generate the following list of test cases:
Why is this needed: Kubernetes operators will quite often take lots of input dimensions into account for their behaviour, all of which should be tested. The static nature of Kuttl at the moment makes it very hard to effectively test a large number of possible input combinations, even though its namespace isolation and parallelization would lend itself very well to quickly test large numbers of scenarios.
This suggestion, if implemented, would allow easily defining a larger test matrix to significantly enhance test coverage.
All of which I have written here is not a complete proposal yet, it is more an interest check, to see if it is worth starting the KEP process for this. If there is positive feedback for this I'll happily add more detail to this and transition it into a proper KEP.