Test cascade

[LupusTheCanine]

Is your feature request related to a problem? Please describe. I started working on a script that implements linked list for fixed length strings. I wanted to implement tests to make validating code easier. My script has extensive internal state that I do not want to expose to other scripts to avoid collisions as well as prevent state coruption. I want to provide a sequence of test inputs that validate that the script is beaving corectly.

Describe the solution you'd like I would like to see a variant of test case that doesn't reset scripts states.

Describe alternatives you've considered Another alternative would be a test case that sends specified sequence of inputs instead of only setting initial state. Another posibility would be allowing internal state of a script to be set. I could also make internal state global but this would interfere with compilation and be error prone.

I think all above solutions are more labor intensive for a script developer than test case sequence.

Additional context

Example code
cases:
  - name: Initial Test
    # optional: global variables to set before running. ':' can be omitted
    stopwhen:
      lm: 0
    inputs:
      wp: "AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHH"
      lm: 5
    # optional: expected value for global variables after running
    # it he values after execution the scipts do not match the values here, the test fails
    outputs:
        lm: 0
        cws: "HHHH"
  - name: backwards no more #here yolol internal state is reset but I would like execution to continue
    stopwhen:
      lm: 0
    inputs:
      lm: -1
    outputs:
      cws: "HHHH"

[dbaumgarten]

That seems reasonable.

Allowing internal variables to be set for tests seems a but messy, after all the internal variables are just a result of previous input. Supplying a sequence of inputs is difficult, because of it's dynamic nature. I guess you could already add a second script to the test that just puts a sequence of inputs into global variables, but thats not really user-friendly.

I guess keeping the state between test-cases should not be too hard (even if there are a few things to get right), and by adding a new field to the test-yaml like "keepstate: true" that behaviour could be enabled when needed.

[LupusTheCanine]

Another thing that comes to my mind that would make providing sequential input would be "maxlines: n" for each test case as well as some way of delaying test end after condition is met. maybe call "finishlines: n" or "finallines: n" where n is a number of lines to be executed after the variable state stop condition is satisfied defaulting to 0 to keep current behavior as default, output would be checked after script is stopped/paused.

[dbaumgarten]

Maxlines for each Test-Case would make sense. That way you could have testcases the do just run a few ticks and don't wait for any variables (effectively creating a delay until the next testcase is started).

[dbaumgarten]

So, I just pushed some changes to the develop-branch, which you can test using this release: https://github.com/dbaumgarten/yodk/releases/tag/latest

You can no set "sequential: true" for the test, which makes every case keep the state from previous cases. Also, there is no a "maxlines" for every test-case, which specifies how much lines to run for that soecific test-case. So you can do things like: "Run case2 for 10 lines, set new inputs with case 2 and run that for 20 lines then check the outputs.

See here and here for examples.

This was more complicated than expected and I just hope I didn't break anything important...

[dbaumgarten]

Published as part of v0.1.10