RFE: pipeline improvements

[smoser]

I have two things I'd like to do in pipelines and would like to not use shell.

1. a generic test executer that would support in a much cleaner way the sort of tests we have in wolfi that basically do program --help | grep ${{pkg.version}}.
2. improve test/daemon-check-output that is in wolfi right now

I can see a path to this where I

1. write a program that does what i want
2. put it in a wolfi package
3. call it from a pipeline that needs: my-testing-package

That would mostly work and would allow me to implement the tests without shell. The limitations that I see with it are:

1. the only way to pass information to a pipeline is via substitution (${{inputs.varname}}).

   In order to invoke 'my-test', I have to have something like:

   pipeline:
   - runs: |
       my-test "${{inputs.var1}}" "${{inputs.var2}}"

   That opens me up to shell problems (what if my inputs have a quote in them?) and limits the inputs to simple scalars.

   The easiest way to solve this would be to put the json or yaml formatted data into the environment when the test is invoked.

2. the input schema is flat key/value. I would like to be able to pass an array or dictionary to the test app.

   I do like that melange checks user input to the pipeline. It'd be nice if we could provide a schema in some way (json schema or the like).

3. I'd like for '--debug' to pass through somehow so that the program can increase verbosity accordingly.

I think if we address those two limitations then we have a pretty good pipeline/test framework.

[smoser]

here is the closest I could do right now.

# daemon-check-output2.yaml
needs:
  packages:
    - my-go-tester

inputs:
  inputblob:
    required: true
    description:
      a correctly formatted yaml or json blob input for mytest

pipeline:
  - name: "run-mytest"
    runs: |
      /usr/bin/daemon-check-output --input=- <<"_BLOBBY_EOF"
      ${{inputs.inputblob}}
      _BLOBBY_EOF

And then I'd use that like:

test:
 pipeline:
  - uses: test/daemon-check-output2
    with:
      inputblob: |
        start: ["env", "USER=FOO", "my-daemon"]
        timeout: 30
        expected_output:
          - my-daemon is running

This would work, but would require the user to put a yaml blob inside 'inputblob'.

[smoser]

Full disclosure, the goal here is to generally separate test running out of melange ... or at least to keep the api very light.

[smoser]

Here is a off-the-cuff example of something i'd like to have

test:
  pipeline:
   - uses: test/program-executer
     with:
       tests:
        - name: "Execute --help"
          execute: [whizbang, --help]
          expect:
            rc: 0
            stdout-match: version.*${{package.version}}
            stderr: null

        - name: "execute cat"
          execute: ["cat"]
          stdin: "hello world"
          expect:
            rc: 0
            stdout: "hello world"

program-executor would then just parse the yaml and start executing tests.

I see this as a more declarative version of the "whizbang --help | grep version" tests that we have.

I'm happy to be told of a nicer way to do this, or an already existing way.