Create output files for automated testing

jmcook1186 commented 3 months ago

What

Create a folder of output files that can be used for automated testing using if-check

Why

We currently have a set of manifests that can be used for manual testing. But we will soon have if-check available that will be used for automated tests in our CI/CD.

In order to do this, we need to reorganize our manifests so that:

we have output files, not pre-execution manifests (because if-diff will compare the given outputs to re-executed outputs
we separate out our output files to be used for automated testing from manifests used as examples/templates for users

@MariamKhalatova I opened a branch (reorg-manifest-folder) where I moved our manifest around into the following folders:

manifests
  |- examples
  |        |-builtins
  |        |-pipelines
  |
  |-outputs
  |        |-bugs
  |        |-features
  |        |-pipelines
  |        |-plugins

Now what we need to do is execute all the manifests. For each manifest in examples we just want to confirm that the examples execute successfully -if so, move on, if not, fix them. For each manifest in outputs we want to overwrite the manifest with its post-execution output file.

Context This will help support automated testing and supporting users with examples and sharing pipelines

Prerequisites/resources no

SoW (scope of work)

[x] manifests file reorganized
[ ] manifests in examples executed and confirmed to run successfully
[ ] manifests in outputs replaced with their post-execution output file

Acceptance criteria

[ ] manifests reorganized into clear structure that separates out examples from outputs for automated tests
[ ] all manifests execute as expected

zanete commented 3 months ago

@jmcook1186 and @MariamKhalatova to sync up and work on this together as @jmcook1186 would like to decide the file structure before any work is done