Nice to have in the DSL

[hsanjuan]

I'm going to keep track here of a few things that cross my mind and that would be nice to have:

• Sucesses should be implicit and it should be expected that everything succeeds automatically
• Should stop testing if a test fails
• Should provide a way to keep an env per Pod so that when running in parallel in a bunch of nodes save_to SOMETHING in a pod does not overwrite the env for everyone else
• Should allow to run everywhere (i.e. on_node: -1)
• Should include networking operations support (i.e. cut network from pod X)
• Should include custom post-exec commands when auto-scaling
• Should allow to define grace time after each step
• Should allow to work with stderr also
• Should be able to run in quiet mode with only test results

[dgrisham]

It'd be nice to be able to do something to the effect of

input:
    - HASH_[1..10]

rather than

input:
    - HASH_1
    - HASH_2
    [...]
    - HASH_10

---

• Should have a topology option that makes it easy to set up certain connections (e.g. 'all nodes disconnect from current peers and establish direct connections with one another)
• Should have some sort of looping mechanism, e.g. for a Bitswap test where two nodes exchange data back-and-forth and output bitswap ledgers. Maybe a clean way to do this is just combine tests, e.g. one test that runs once and sets up topology/initial conditions, then a second test that's set to run N times does the bitswap exchanges and outputs ledgers.
• Could be nice to have a variable that defaults to the current node number, for something like (current node number stored in i):

step: Save LANADDR for all nodes
on_node: 1
end_node: 10
cmd: "ipfs id -f "<addrs>" | grep 172
outputs:
   - line: 0
     save_to: LANADDR$i

One way to implement this might be to have a 'global' environment, which includes environment variables that are available both outside of cmds and also 'inside' (within the containers themselves).

[ZenGround0]

A few of the above suggestions have been addressed in some form in recent additions to the DSL.

• Running tests everywhere can now be readily specified by setting the selection field in a test to run over the range of all test node ids or over 100 percent of the nodes
• Cutting the network from pod has been tricky to achieve with kubernetes alone. The cluster application gets around the issue by setting up daemon processes in the background of its testing container image so that tests can send STOP or KILL signals to the process to take it out of commission. A more general solution would still be very worthwhile but I think would need to involve a customized kubernetes controller.
• There is now a looping mechanism that will iterate a given number of times, or will perform a for-each iteration over an input array
• I am not 100 % on the context of the HASH [1..10] suggestion, but it is perhaps addressed by the recent addition of the append_to output and automatic inclusion of these arrays into the environment of commands that make accesses.
• The current node number can be specified as "%s", while the current iteration number can be specified as "%i". Currently these sequences are only interpreted as such when the command makes an array access, e.g. ${OUT_CIDS[%s]}. It should be straightforward to generalize this substitution to other places within the command.

[ZenGround0]

It would be nice to account for basic arithmetic when making substitutions during the parameterization pre-processing step. For example the current cluster-test runner script needs to write 13 parameters to the config in order to substitute in all the necessary parameters. However all 13 parameters are derivable from two root parameters by addition, multiplication and division by constants (e.g. N: 4, Y: 3, N_times_4: 16, ParamA: 2*N*Y + Y/3...). Only including addition, multiplication and parentheses ordering would make parameterizing a lot cleaner.