Add support for running an individual test

[paulmillar]

Motivation:

Sometimes, typically when debugging a specific problem, it is useful to run just a single test from the test-suite. This allows for more extensive debugging to be enabled.

Modification:

Add an optional second command-line argument to 'run-testsuite.sh' that limits the testing to a specific test.

When only a single command-line argument is provided then the tests are run as before.

Result:

It is now easy to run a single test from the suite.

[andreaceccanti]

I typically add a dbg label to a test and set ROBOT_ARGS="-i dbg", since I don't like using the test name as it requires quoting.

You could use ROBOT_ARGS="--name \"test name\"" as well, but probably the approach you propose is simpler for the user.

[paulmillar]

Hiya,

The dbg label is a good hint, but has the (minor) drawback of requiring one to edit the test suide.

As you say, one might expect to be able to use ROBOT_ARGS.

I started off going down that line, attempting with commands like:

ROBOT_ARGS="-t \"test name\"" run-testsuite.sh

It took an embarrassingly long time to figure out why this doesn't work (hint: eval) -- partly from a somewhat cryptic error message and partly because it works perfectly fine with running the robot command on the command-line.

Since getting ROBOT_ARGS environment variable to work with quoted arguments required patching the run-testsuite.sh script, I thought I'd add something a bit nicer: using ROBOT_ARGS is a faff after all.

[giacomini]

Some time ago I did something similar for the VOMS testsuite. The most relevant commit is https://github.com/italiangrid/voms-testsuite/commit/91b8a5cd0aabe14a1170f352c074b989f3a7c7e9. It would be a more general solution for passing arguments from the command line of run-testsuite.sh down to robot. What do you think?

[paulmillar]

Hi all.

@giacomini That would work for me, too.

I'd have thought running a single test is a sufficiently common desire that having it as a special case would make sense; however, I'm happy with whichever solution allows me to run a single test.

Aside: arrays are (IIRC) definitely a bash-ism, so taking the script out of the realm of POSIX shell. I have no problem with this, just mentioning it just in case it matters.

Cheers, Paul.

[giacomini]

I have just created PR #24. Would you please have a look?

For example, now one can do:

[test@d9dffc4ffe20 test-suite]$ ./run-testsuite.sh se-cnaf-amnesiac-storm --test "Token with correct *"        
JWT compliance test suite run against: se-cnaf-amnesiac-storm
==============================================================================
se-cnaf-amnesiac-storm                                                        
==============================================================================
se-cnaf-amnesiac-storm.Audience                                               
==============================================================================
Token with correct audience is accepted                               | PASS |
------------------------------------------------------------------------------
Token with correct audience in multiple option is accepted            | PASS |
------------------------------------------------------------------------------
se-cnaf-amnesiac-storm.Audience                                       | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
se-cnaf-amnesiac-storm                                                | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
Output:  /home/test/test-suite/reports/se-cnaf-amnesiac-storm/output.xml
Log:     /home/test/test-suite/reports/se-cnaf-amnesiac-storm/log.html
Report:  /home/test/test-suite/reports/se-cnaf-amnesiac-storm/report.html

[giacomini]

Superseded by https://github.com/indigo-iam/wlcg-jwt-compliance-tests/pull/24