search

redhat-openshift-ecosystem / openshift-preflight

Operator and container preflight certification tests
Apache License 2.0
60 stars 65 forks source link

Could we have time exposed in results-junit.xml to be in seconds as required by junit format? #324

Closed tkrishtop closed 3 years ago

tkrishtop commented 3 years ago

Bug Description

Junit format requires time to be in seconds, but preflight is using the minute-second format in results-junit.xml.

<testsuite tests="4" failures="2" time="2m52.841440984s" name="Red Hat Certification">

We have time="2m52.841440984s" but, according to junit format, we should have time="172.841440984".

That causes a bug in the DCI pipeline display: duration = 0s.

Version and Command Invocation

preflight v1.0.2

Steps to Reproduce:

1) run preflight operator check 2) results-junit.xml contains time in minute-second format instead of seconds

<testsuites>
    <testsuite tests="4" failures="2" time="2m52.841440984s" name="Red Hat Certification">
        <properties></properties>
        <testcase classname="quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9" name="ScorecardBasicSpecCheck" time="6.802355119s">Check to make sure that all CRs have a spec block.</testcase>
        <testcase classname="quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9" name="DeployableByOLM" time="2m39.18758888s">Checking if the operator could be deployed by OLM, and images are from approved sources</testcase>
        <testcase classname="quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9" name="ScorecardOlmSuiteCheck" time="6.676499075s">
            <failure message="Failed" type="">Check ScorecardOlmSuiteCheck encountered an error. Please review the artifacts/operator_bundle_scorecard_OlmSuiteCheck.json file for more information.: Suggested Fix: See scorecard output for details, artifacts/operator_bundle_scorecard_OlmSuiteCheck.json</failure>
        </testcase>
        <testcase classname="quay.io/rh-nfv-int/testpmd-operator-bundle:v0.2.9" name="ValidateOperatorBundle" time="174.99791ms">
            <failure message="Failed" type="">Check ValidateOperatorBundle encountered an error. Please review the preflight.log file for more information.: Suggested Fix: Valid bundles are defined by bundle spec, so make sure that this bundle conforms to that spec. More Information: https://github.com/operator-framework/operator-registry/blob/master/docs/design/operator-bundle.md</failure>
        </testcase>
    </testsuite>
</testsuites>

Expected Result

To have time in seconds, i.e. to have time="172.841440984" instead of time="2m52.841440984s"

Actual Result

We have time="2m52.841440984s". It should be time="172.841440984".

tkrishtop commented 3 years ago

Hi, @bcrochet thank you for the fix! I tested it, total elapsed time now looks fine but the time for each unit test is still in the previous format <testcase classname="quay.io/rh-nfv-int/testpmd-operator-bundle@sha256:5e28f883faacefa847104ebba1a1a22ee897b7576f0af6b8253c68b5c8f42815" name="ScorecardOlmSuiteCheck" time="10.342975268s">. Here is the result-junit.xml I got

image