redhat-openshift-ecosystem / openshift-preflight

Operator and container preflight certification tests
Apache License 2.0
60 stars 65 forks source link

`check container` does not always write `elapsed_time` to results.json #438

Closed acornett21 closed 2 years ago

acornett21 commented 2 years ago

Bug Description

When running check container not all checks write out an elapsed_time to the results.json. This means that pyxis rejects our request.

Version and Command Invocation

main

Steps to Reproduce:

1) run check container --submit 2) see error returned by pyxis 3)

Expected Result

It should be expected that this value is always present for any check, just like check operator.

Actual Result

results.json does not have elapsed_time for every check.

actual json

{
    "image": "quay.io/opdev/simple-demo-operator:latest",
    "passed": false,
    "test_library": {
        "name": "github.com/redhat-openshift-ecosystem/openshift-preflight",
        "version": "unknown",
        "commit": "unknown"
    },
    "results": {
        "passed": [
            {
                "name": "BasedOnUbi",
                "description": "Checking if the container's base image is based upon the Red Hat Universal Base Image (UBI)",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            },
            {
                "name": "HasUniqueTag",
                "elapsed_time": 930,
                "description": "Checking if container has a tag other than 'latest', so that the image can be uniquely identified.",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            },
            {
                "name": "LayerCountAcceptable",
                "description": "Checking if container has less than 40 layers.  Too many layers within the container images can degrade container performance.",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            },
            {
                "name": "HasRequiredLabel",
                "description": "Checking if the required labels (name, vendor, version, release, summary, description) are present in the container metadata.",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            },
            {
                "name": "HasNoProhibitedPackagesMounted",
                "elapsed_time": 28,
                "description": "Checks to ensure that the image in use does not include prohibited packages, such as Red Hat Enterprise Linux (RHEL) kernel packages.",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            },
            {
                "name": "RunAsNonRoot",
                "description": "Checking if container runs as the root user because a container that does not specify a non-root user will fail the automatic certification, and will be subject to a manual review before the container can be approved for publication",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            }
        ],
        "failed": [
            {
                "name": "RunnableContainer",
                "elapsed_time": 11120,
                "description": "Checking if container runs within a pre-configured timeframe",
                "help": "Check RunnableContainer encountered an error. Please review the preflight.log file for more information.",
                "suggestion": "Ensure that the container can be launched",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            },
            {
                "name": "HasLicense",
                "description": "Checking if terms and conditions applicable to the software including open source licensing information are present. The license must be at /licenses",
                "help": "Check HasLicense encountered an error. Please review the preflight.log file for more information.",
                "suggestion": "Create a directory named /licenses and include all relevant licensing and/or terms and conditions as text file(s) in that directory.",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            }
        ],
        "errors": [
            {
                "name": "RunSystemContainer",
                "elapsed_time": 171,
                "description": "Checking if a container can run as a systemd service",
                "help": "Check RunSystemContainer encountered an error. Please review the preflight.log file for more information.",
                "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
                "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
            }
        ]
    }
}

Additional Context

Response from pyxis

{
  "detail": {
    "passed": [
      "Missing data for required field."
    ],
    "results": {
      "failed": {
        "1": {
          "elapsed_time": [
            "Missing data for required field."
          ]
        }
      },
      "passed": {
        "0": {
          "elapsed_time": [
            "Missing data for required field."
          ]
        },
        "2": {
          "elapsed_time": [
            "Missing data for required field."
          ]
        },
        "3": {
          "elapsed_time": [
            "Missing data for required field."
          ]
        },
        "5": {
          "elapsed_time": [
            "Missing data for required field."
          ]
        }
      }
    }
  },
  "status": 400,
  "title": "Bad Request",
  "type": "about:blank",
  "trace_id": "0x64d2f4e1488c50125260d6ecc39da1c5"
}
acornett21 commented 2 years ago