Capturing failing tests

[neomatrix369]

Adding check for failing tests when a negative condition is induced, applying this check to Java and Scala, others to follow soon.

Continuation of from #33 and #32.

[neomatrix369]

Scala test results

[snipped]
Test passed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test is expected to pass, as we are expecting a non-zero exit code returned by the docker container execution and an empty result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=scala|challenge=SUM|exitCode=0|expected=100|actual=100
  language=scala|challenge=TST|exitCode=0|expected=50|actual=50
  language=scala|challenge=CHK|exitCode=0|expected=0|actual=0
  language=scala|challenge=xyz|exitCode=10|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

Java test results:

[snipped]
Test passed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test is expected to pass, as we are expecting a non-zero exit code returned by the docker container execution and an empty result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=java|challenge=SUM|exitCode=0|expected=100|actual=100
  language=java|challenge=TST|exitCode=0|expected=50|actual=50
  language=java|challenge=CHK|exitCode=0|expected=0|actual=0
  language=java|challenge=xyz|exitCode=10|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

Hmmm tests

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=hmmm|challenge=SUM|exitCode=0|expected=33|actual=33
  language=hmmm|challenge=SUM|exitCode=0|expected=44|actual=44
  language=hmmm|challenge=SUM|exitCode=0|expected=55|actual=55
  3 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

Just tried removing || true from python and it's not a good idea I think. So pushed a modified version and this is the result:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test is expected to pass, as we are expecting a non-zero exit code returned by the docker container execution and an empty result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=python|challenge=SUM|exitCode=0|expected=100|actual=100
  language=python|challenge=TST|exitCode=0|expected=50|actual=50
  language=python|challenge=CHK|exitCode=0|expected=0|actual=0
  language=python|challenge=xyz|exitCode=10|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

We cannot do anything to NodeJS, as coverage process fails if we remove || true.

Also unable to add the below to our container tests:

checkForFailingCoverageResults "nodejs" "master" "xyz" "" 

Everything returns a zero exit code, failed or not.

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=nodejs|challenge=SUM|exitCode=0|expected=100|actual=100
  language=nodejs|challenge=TST|exitCode=0|expected=75|actual=75
  language=nodejs|challenge=CHK|exitCode=0|expected=0|actual=0
  3 test(s) passed

  ~~~ Failed Tests ~~~
  language=nodejs|challenge=xyz|exitCode=0|expected=|actual=0
  1 test(s) failed

[neomatrix369]

CSharp, FSharp and VBNet, cannot be run with || true which means we need to find another way out of this

[neomatrix369]

Issue raise #35 to deal with languages that do not throw a non-zero exit code when they fail

[neomatrix369]

Got ruby tests passing:

[snipped]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test is expected to pass, as we are expecting a non-zero exit code returned by the docker container execution and an empty result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=ruby|challenge=SUM|exitCode=0|expected=100|actual=100
  language=ruby|challenge=TST|exitCode=0|expected=83|actual=83
  language=ruby|challenge=CHK|exitCode=0|expected=0|actual=0
  language=ruby|challenge=xyz|exitCode=1|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

Nodejs tests pass:

[snipped]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test is expected to pass, as we are expecting a non-zero exit code returned by the docker container execution and an empty result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=nodejs|challenge=SUM|exitCode=0|expected=100|actual=100
  language=nodejs|challenge=TST|exitCode=0|expected=75|actual=75
  language=nodejs|challenge=CHK|exitCode=0|expected=0|actual=0
  language=nodejs|challenge=xyz|exitCode=1|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

CSharp tests now work:

[snipped]
Test passed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test is expected to pass, as we are expecting a non-zero exit code returned by the docker container execution and an empty result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=csharp|challenge=SUM|exitCode=0|expected=100|actual=100
  language=csharp|challenge=TST|exitCode=0|expected=50|actual=50
  language=csharp|challenge=CHK|exitCode=0|expected=0|actual=0
  language=csharp|challenge=xyz|exitCode=1|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

FSharp tests

[snipped]
~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=fsharp|challenge=SUM|exitCode=0|expected=100|actual=100
  language=fsharp|challenge=TST|exitCode=0|expected=50|actual=50
  language=fsharp|challenge=CHK|exitCode=0|expected=0|actual=0
  language=fsharp|challenge=xyz|exitCode=1|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

Tada! VBNet tests:

[snipped]
Test passed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test is expected to pass, as we are expecting a non-zero exit code returned by the docker container execution and an empty result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ Summary of test executions ~~~
  ~~~ Passed Tests ~~~
  language=vbnet|challenge=SUM|exitCode=0|expected=100|actual=100
  language=vbnet|challenge=TST|exitCode=0|expected=50|actual=50
  language=vbnet|challenge=CHK|exitCode=0|expected=0|actual=0
  language=vbnet|challenge=xyz|exitCode=1|expected=|actual=
  4 test(s) passed

  ~~~ Failed Tests ~~~
  0 test(s) failed

[neomatrix369]

Lets discuss the tdl-runner changes for ruby, nodejs, and the .net variants - to see if we can do something more to make sure we have covered more ground.