Azure / simdem

Tool for Simulating Demo's, delivering Tutorials and using documentation as tests.
MIT License
34 stars 17 forks source link

SimDem 2 Feature: Setup/Teardown #96

Closed lastcoolnameleft closed 6 years ago

lastcoolnameleft commented 6 years ago

For tests to be run over again, some walkthroughs need to clean up after themselves. But not completely.

For example, if I want to demo deploying a Helm chart, once the main section is done, we want to allow the option to clean up after itself and delete the helm chart. However, we don't want it to delete the full K8S instance.

We also might want to enable performing a clean prior to running. (i.e. Similar to setup/teardown of most test frameworks)

This gets awkward when factoring in Prerequisites (which have validations to determine if they should be run). My initial thought is to have them independent, and keep Prerequisites focused on conditionals/validation.

My proposal is to introduce 2 optional sections to markdown:

SorraTheOrc commented 6 years ago

The way I work this in the current implementation is to use separate markdowns for each and reference setup steps in the prerequisites and teardown in the next steps. This works well for tutorials and manually executed demos, but less well for automated testing (which doesn't run next steps).

To facilitate cleanup in automated tests I implemented a "test plan" feature, see docs and example. It turns out that there isn't actually a need for the separate parsing of a test plan anymore (it can be done in mardown now).

This is the main driver for having a validation section for pre-reqs. It enables us to run the setup steps (defined in prerequisites) then cleanup some of the work for the next test but only have the necessary prerequisites re-run.

In all the cases I've built so far this works well and I don't see the need for explicit new sections in the doc. Can you give a concrete example where they are needed and why.

lastcoolnameleft commented 6 years ago

Implementing a separate test-plan file is doable, but IMO it is more valuable from a documentation perspective to have everything in one file.

The scenario I want to implement is:

The last step is necessary so that I can re-run the test. Of course, the validation of the service should go in the validation section, but I believe the removal of the helm chart needs it's own section (or perhaps file) so that I can re-run the test repeatedly.

lastcoolnameleft commented 6 years ago

After exploring an implementation, here's what I've tried and like:

Now if you want to run a test + cleanup, you can:

simdem -m test README.md
simdem -m cleanup README.md

Thoughts?

SorraTheOrc commented 6 years ago

I'm +0 on your proposal. I've found the separate test plan file to be invaluable as it allows steps to be reused across multiple different documents. I'm a big fan of reuse.

If you prefer your approach, which has some advantages, then go ahead an implement it, it is compatible with my own approach.

lastcoolnameleft commented 6 years ago

Good to know it won't break anything. It was a pretty minor update to implement and for right now I'd like to experiment with it.

https://github.com/Azure/simdem/commit/efeb0bd2a906ecc37c34bc677994087fe42f670d

Closing since there's no issues and it's implemented.