search

operator-framework / ansible-operator-plugins

Experimental extraction/refactoring of the Operator SDK's ansible operator plugin
Apache License 2.0
7 stars 17 forks source link

e2e-ansible-molecule test improvements #21

Open asmacdo opened 2 years ago

asmacdo commented 2 years ago

The ansible-molecule GH action takes on average 35 minutes to run on CI, which is more than double the test time for the other e2e tests and is a bottleneck for the whole project. I recently went through these tests and collected a handful of improvements to make. The goal would be to bring the ansible-molecule test runtime down to about 15 minutes to be on-par with go and helm e2e.

[ ] Asynchronous setup We create each object and wait for it to finish before we create the next. This change brings the runtime down from 35 to 27 minutes https://github.com/operator-framework/operator-sdk/pull/5679

[ ] Separate memcached ansible molecule test into a separate action We run 2 sets of tests (in serial), advanced-molecule-operator and memcached-ansible-molecule. The tests in advanced-molecule-operator are our primary ansible tests. memcached-ansible-molecule also runs some tests that could be consolidated into advanced-molecule-operator with the exception of the cluster setup. memcached-ansible-molecule is designed to be similar to how operator authors might test their operators, and so uses the kind scenario within molecule. This scenario creates a kind cluster. ansible molecule e2e tests therefore create a cluster, deploy operators and test them, destroys the cluster, creates a second cluster, deploys operators and tests them and destroys the cluster. There is no shared state, and separating this component could probably cut the runtime down by half.

[ ] Refactor the test re-generation The main difficulty is that the test operators are scaffolded from scratch each time in order to keep up with operator-sdk updates, and then all test code is inserted. This would make sense for files that the operator-sdk actively changes, but the majority of the changes are to playbooks, molecule verify tasks, and the watches file. So rather than searching for strings and inserting code, I propose that we simply checkin the playbooks, molecule verify tasks, and the watches file for each of the test operators, and then clobber them. For playbooks and verify tasks, we will be replacing nothing. The watches file can be generated by api creation, but does not include extras like finalizers. This change will bring the testing requirements closer to the requirements of operator-authors, who also have to frequently upgrade operator-sdk version.

camilamacedo86 commented 2 years ago

Could we begin to have the molecule tests scaffolded by default under the testdata? WDYT?

asmacdo commented 2 years ago

@camilamacedo86, do you mean committing them to the repo? They are currently scaffolded there at test runtime, but not committed to the repo. If so, big +1

If I misunderstood could you elaborate?

openshift-bot commented 1 year ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

camilamacedo86 commented 1 year ago

/remove-lifecycle stale /lifecycle frozen