search

voxpupuli / onceover

Your gateway drug to automated infrastructure testing with Puppet
Apache License 2.0
142 stars 45 forks source link

Code Coverage Reports #184

Closed EpicPhil closed 5 years ago

EpicPhil commented 5 years ago

Is there a way to get coverage reports for our onceover tests?

dylanratcliffe commented 5 years ago

Possibly, but the code coverage tools that exist wouldn't work, here's why:

Onceover is designed to work with the roles and profiles pattern, in which each machine gets only one role, which includes many profiles, which call out to forge modules, which do the work. Onceover only tests roles, not profiles (normally) so if you were to look at the code coverage it would be low. However given that each node only gets one role, and nothing should be getting profiles assigned directly to it (only through a role) this means that if you test every role you essentially have 100% coverage, because each node only gets one role and that's it meaning all permutations are covered.

Does that make sense?

EpicPhil commented 5 years ago

Yeah, thanks for the response.

I currently have onceover testing roles to make sure they include the expected profiles with the appropriate parameters, and I also test the profiles themselves to make sure their sub-resources are called correctly. I was thinking of the logic I have in my profiles for executing different code paths based on profile parameters or hiera data, and making sure I covered all those branches.

dylanratcliffe commented 5 years ago

That's true, but you'd have to certainly modify the way the existing code coverage tools work as they would be expecting a test for every resource. To be honest I'd argue that testing for roles to contain certain profiles is just a duplication of effort, unless you want to globally enforce that "profile::base" included in every role or something similar. As for profiles I'd also argue that unless a resource is wrapped in some logic that you want to make sure works, testing for them is redundant, it just creates a second place you have to change code if you ever refactor it.

I think a really interesting code coverage report would involve not classes that don't have tests, but classes which exist in the repo (in the controlrepo, not in modules) where not compiled as part of any test. That would make it very easy to ensure you had enough factsets to cover all permutations of your if-else logic etc.

dylanratcliffe commented 5 years ago

Closing due to inactivity