search

techthoughts2 / Catesta

Catesta is a PowerShell module and vault project generator. It uses templates to rapidly scaffold test and build integration for a variety of CI/CD platforms.
https://www.catesta.dev/
MIT License
169 stars 20 forks source link

Build before testing/analysis #3

Closed ChrisLGardner closed 6 months ago

ChrisLGardner commented 4 years ago

Description

If you test the individually and then build the module you aren't accurately testing the code that you're shipping, simple things like $script: or $global: variables may be set in functions that break interactions in other functions but pass the tests. There are also other more complex interactions that can occur with the dot sourcing process (assuming the individual function tests are importing the psd1 or psm1 rather than just the individual function file) which don't occur when all the functions are in a single file.

Describe the solution you'd like

Have the test and analyze steps depend on the build step. It takes a tiny amount of time to combine the ps1 files into a psm1 so it's well worth doing it first.

Additional context

As a side benefit this will also speed up tests which import the module with -Force since dot sourcing is really slow. It'll not be noticeable for small modules but for bigger ones or ones with lots of tests that many imports will have an impact on test time.

techthoughts2 commented 4 years ago

If you test the individually and then build the module you aren't accurately testing the code that you're shipping, simple things like $script: or $global: variables may be set in functions that break interactions in other functions but pass the tests.

This is solved with an Imports file: https://github.com/techthoughts2/Catesta/blob/master/src/Catesta/Resources/Module/src/Module/Module.psm1#L4

It permits availability of all script or globally scoped variables.

Have the test and analyze steps depend on the build step.

There is a drawback to this approach. The test function needs to import the module. In current form, that imports the de-constructed psm1. The benefit to this is that you get the ability to run individual pester tests against one function. If your change were implemented, the pester function test would need to be updated to reference the build location. That means you would have to run a build locally every time you wanted to test a function during development.

For unit testing I feel like that is not a trade-off worth taking. I'm open to discussing if you have an idea of supporting both scenarios.

I do agree that you make a valid point for infra tests. I will pop a new issue to have infra tests moved to post build.

ChrisLGardner commented 4 years ago

Have the test and analyze steps depend on the build step.

There is a drawback to this approach. The test function needs to import the module. In current form, that imports the de-constructed psm1. The benefit to this is that you get the ability to run individual pester tests against one function. If your change were implemented, the pester function test would need to be updated to reference the build location. That means you would have to run a build locally every time you wanted to test a function during development.

For unit testing I feel like that is not a trade-off worth taking. I'm open to discussing if you have an idea of supporting both scenarios.

I do agree that you make a valid point for infra tests. I will pop a new issue to have infra tests moved to post build.

Ensuring the Tests import the module is pretty easy, just add the build output location to the beginning of your PSModulePath as part of the build process and you'll always import that version first. That way you don't care about where the module is output and you can test the whole thing easily.

As a way to test individual functions, my approach for that has been to tag the Describe blocks for each test script with the name of the function it's testing, that way I can run my build script with a -Tag parameter and just run the functions I want to. Building the module takes 5 - 10 seconds usually so I'm willing to accept that small amount of time being added to my local build process, especially when I'm usually just hitting F5 in VSCode to trigger it. And running a build locally is how other programming languages have to do it and some of those take a lot longer to build, it's not a huge trade off to ensuring that you've got the latest version of the code in your module files, especially if you then go on to do some manual exploratory testing.

If you test the individually and then build the module you aren't accurately testing the code that you're shipping, simple things like $script: or $global: variables may be set in functions that break interactions in other functions but pass the tests.

This is solved with an Imports file: https://github.com/techthoughts2/Catesta/blob/master/src/Catesta/Resources/Module/src/Module/Module.psm1#L4

It permits availability of all script or globally scoped variables.

That was just one example of possible issues with this approach. There's also the length of time it takes to import dot sourced modules, it's not a huge amount of time but can get to 1 - 2 seconds easily per import. Not an issue for an end user usually but if you're running 10+ test scripts it adds up and anything that can be done to speed up the build and test process is well worth it.