search

juju-solutions / matrix

Automatic testing of big software deployments under various failure conditions
Other
8 stars 9 forks source link

Glitch should only apply changes to apps deployed by the test #24

Closed johnsca closed 7 years ago

johnsca commented 7 years ago

While we should generally run matrix tests on clean models, for dev / testing it's often helpful to (re)run against a pre-deployed model. However, if the current model includes applications that are not part of the matrix test, we might cause mutations to them which are not relevant to the test.

I think glitch should start from context.apps instead of context.juju_model.applications.

pengale commented 7 years ago

I've actually been taking advantage of the fact that matrix will go to town on stuff by deploying things by hand in advance, and then running just the glitch bits of matrix.

That was largely a workaround for when the deploy bits of matrix weren't stable, though. I suppose that need will go away with the actual production version of matrix.

johnsca commented 7 years ago

I should clarify; as long as the apps are mentioned in the bundle, they will be added to context.apps, even if they were pre-deployed. This should already work, even with the issues with libjuju.

My concern is that the tests will be run with a specific set of charms in mind, and if there are other, unrelated apps in the environment, we should ignore them. Currently, the health check only checks apps in the context.apps list. I suppose another option would be to make all tasks look at the model as a whole, but that seems a bit messier.

pengale commented 7 years ago

@johnsca Aha. Got it. That makes a lot of sense.

pengale commented 7 years ago

@johnsca Is this addressed to your satisfaction by https://github.com/juju-solutions/matrix/pull/43?

johnsca commented 7 years ago

Yep.