quattor / maven-tools

Maven-based Build Tools
www.quattor.org
Apache License 2.0
3 stars 12 forks source link

junit / xunit issue #181

Closed stdweird closed 6 years ago

stdweird commented 6 years ago

(this if for documentation purposes only; there's no issue with maven-tools)

Jenkins runs the unittests with the perl TAP::Harness enabled, and with TAP::Harness::Junit installed. it produces a junit_output.xml file that is then parsed by the xUnit plugin to publish a report.

however, xUnit seems to have some issue with the time format and fails the test

INFO: Processing JUnit
INFO: [JUnit] - 1 test report file(s) were found with the pattern 'junit_output.xml' relative to '/var/lib/jenkins/jobs/CCM/workspace' for the testing framework 'JUnit'.
WARNING: The file '/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml' is an invalid file.
WARNING: At line 3 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '3.80636811256409' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '3.80636811256409' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 10 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '6.05227303504944' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 10 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '6.05227303504944' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 642 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '60.5245859622955' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 642 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '60.5245859622955' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 1926 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '5.59209990501404' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 1926 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '5.59209990501404' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 1967 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '0.640208959579468' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 1967 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '0.640208959579468' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 2230 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '4.0312340259552' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 2230 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '4.0312340259552' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 2269 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '3.86692810058594' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 2269 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '3.86692810058594' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 2338 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '1.42449498176575' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 2338 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '1.42449498176575' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 2499 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '5.50598907470703' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 2499 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '5.50598907470703' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 2864 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '1.4761791229248' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 2864 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '1.4761791229248' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 3031 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '1.41334986686707' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3031 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '1.41334986686707' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 3126 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '4.38200998306274' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3126 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '4.38200998306274' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 3197 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '10.7618389129639' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3197 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '10.7618389129639' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 3554 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '0.379251956939697' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3554 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '0.379251956939697' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 3564 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '1.36555409431458' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3564 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '1.36555409431458' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 3653 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '5.47005200386047' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3653 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '5.47005200386047' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 3996 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '1.33711576461792' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3996 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '1.33711576461792' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4037 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '3.87281513214111' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4037 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '3.87281513214111' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4518 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '1.57949805259705' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4518 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '1.57949805259705' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4539 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '0.354336023330688' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4539 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '0.354336023330688' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4575 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '4.46588397026062' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4575 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '4.46588397026062' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4698 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '0.614604949951172' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4698 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '0.614604949951172' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4719 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '0.667536020278931' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4719 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '0.667536020278931' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4798 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '7.43401193618774' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4798 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '7.43401193618774' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
WARNING: At line 4967 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-pattern-valid: Value '0.466110944747925' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 4967 of file:/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml:cvc-attribute.3: The value '0.466110944747925' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
ERROR: The plugin hasn't been performed correctly: The result file '/var/lib/jenkins/jobs/CCM/workspace/junit_output.xml' for the metric 'JUnit' is not valid. The result file has been skipped.
Build step 'Publish xUnit test result report' changed build result to FAILURE

xUnit 2.0.4 has an issue fixed https://issues.jenkins-ci.org/browse/JENKINS-52173 that spits out the same errors, but it doesn't fix our unit tests (even with 2.1.0).

stdweird commented 6 years ago

following patch was applied to the TAP::Harness::Junit module (which was installed using cpan, and is local to jenkins) to force the reported time output to 3 digits after comma, to make the junit_output.xml "valid" again. TAP::Harness::Junit hasn't seen any development in years, so i think this patch is safe; and i also assume we can't upstream the patch as it way too specific (and feels like it works around some xunit issue about xunit being way to restricted)

--- perl/lib/perl5/TAP/Harness/JUnit.pm.orig    2018-07-20 12:47:24.055481018 +0200
+++ perl/lib/perl5/TAP/Harness/JUnit.pm 2018-07-20 12:49:41.135478156 +0200
@@ -199,6 +199,7 @@

    my $time = $parser->end_time - $parser->start_time;
    $time = 0 if $self->{__notimes};
+   $time = sprintf("%.3f", $time);

    # Get the return code of test script before re-parsing the TAP output
    my $badretval = $parser->exit;
@@ -243,6 +244,7 @@

        my $time = $result->{__end_time} - $result->{__start_time};
        $time = 0 if $self->{__notimes};
+       $time = sprintf("%.3f", $time);

        # Counters
        if ($result->type eq 'plan') {
stdweird commented 6 years ago

@jrha @jouvin @ned21 tests shouldn't be failing anymore (but you might need to rerun them)

jrha commented 6 years ago

@stdweird thanks for digging into this and sorting it out