define return criteria for test cases

[garypicher]

Test cases that run as part of our regression build need to adhere to a defined return code criteria so that the test framework can aggregate test results and show pass / fail status. A simple mechanism might be to return 0 for a passing test and a 1 for a failing test. In theory, we could say 0 for a passing test and 'anything else' for a failing test, however that doesn't give us flexibility to define additional return codes later. I can't think of a use for additional return codes right now, but you never know...

[c1505]

Ultimately all of the test data is translated to junit style xml . Currently there are only bash test scripts that return a zero for failure as defined here https://github.com/kabanero-io/kabanero-regression-testing/blob/master/maintest.sh . I can see people wanting to write tests in other languages potentially with test frameworks in the future. For each new test language and/or framework used, we will need to define the way it is translated into the junit style xml. I think for the current shell scripts a return code of zero to then translate into the junit failure is appropriate.