search

redhat-openshift-ecosystem / openshift-preflight

Operator and container preflight certification tests
Apache License 2.0
60 stars 65 forks source link

[ScorecardOlmSuiteCheck] File operator_bundle_scorecard_OlmSuiteCheck.json is absent #307

Closed tkrishtop closed 2 years ago

tkrishtop commented 3 years ago

Bug Description

File operator_bundle_scorecard_OlmSuiteCheck.json mentioned in results.json is absent in /artifacts

Version and Command Invocation

The latest preflight master + operator-sdk latest master inside: docker.io/sbarouti/preflight:operator-sdk

Steps to Reproduce:

1) The command to run:

podman run \
-it \
--rm \
--pull=always \
--name preflight \
--privileged \
-e PFLT_JUNIT=true \
-e PFLT_ARTIFACTS=/artifacts \
-e PFLT_LOGFILE=/artifacts/preflight.log \
-e PFLT_LOGLEVEL=trace \
-e KUBECONFIG=/kubeconfig \
-e PFLT_INDEXIMAGE={{ OO_INDEX }} \
-e PFLT_NAMESPACE=preflight-testing \
-e PFLT_SERVICEACCOUNT=default \
-v {{ preflight_kubeconfig }}/kubeconfig:/kubeconfig \
-v {{ preflight_artifacts }}:/artifacts \
{{ preflight_version }} \
check operator \
{{ operator }}

2) results.json mentions that there is more info in artifacts/operator_bundle_scorecard_OlmSuiteCheck.json.

    "errors": [
      {
        "name": "ScorecardOlmSuiteCheck",
        "elapsed_time": 8230,
        "description": "Operator-sdk scorecard OLM Test Suite Check",
        "help": "Check ScorecardOlmSuiteCheck encountered an error. Please review the artifacts/operator_bundle_scorecard_OlmSuiteCheck.json file for more information."
      },

3) This file is absent in /artifacts

changed: [jumphost] => (item=results.json)
changed: [jumphost] => (item=results-junit.xml)
changed: [jumphost] => (item=preflight.log)
changed: [jumphost] => (item=hashes.txt)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
failed: [jumphost] (item=operator_bundle_scorecard_OlmSuiteCheck.json) => {"ansible_loop_var": "file_to_copy", "changed": false, "file_to_copy": "operator_bundle_scorecard_OlmSuiteCheck.json", "msg": "Could not find or access '/tmp/ansible.xv80kebwpreflight_tmp_dir/artifacts/operator_bundle_scorecard_OlmSuiteCheck.json' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

Expected Result

To have file operator_bundle_scorecard_OlmSuiteCheck.json in /artifacts.

Actual Result

There is no file operator_bundle_scorecard_OlmSuiteCheck.json in /artifacts.

Additional Context

Here are errors in preflight.log

time="2021-10-25T15:59:37Z" level=trace msg="running scorecard with the following invocation[operator-sdk scorecard --output json --selector=suite=olm --kubeconfig /kubeconfig --namespace preflight-testing --service-account default --config /tmp/scorecard-test-config-999783513.yaml --verbose /tmp/preflight-2224823739/fs]"
time="2021-10-25T15:59:45Z" level=error msg="stdout: {\n  \"kind\": \"TestList\",\n  \"apiVersion\": \"scorecard.operatorframework.io/v1alpha3\",\n  \"items\": [\n    {\n      \"kind\": \"Test\",\n      \"apiVersion\": \"scorecard.operatorframework.io/v1alpha3\",\n      \"spec\": {\n        \"image\": \"quay.io/operator-framework/scorecard-test@sha256:4e166f5a68a5183986d512d8647c0da9436276d5c1c8ed7ab31cf19ed5a1649a\",\n        \"entrypoint\": [\n          \"scorecard-test\",\n          \"olm-bundle-validation\"\n        ],\n        \"labels\": {\n          \"suite\": \"olm\",\n          \"test\": \"olm-bundle-validation-test\"\n        },\n        \"storage\": {\n          \"spec\": {\n            \"mountPath\": {}\n          }\n        }\n      },\n      \"status\": {\n        \"results\": [\n          {\n            \"name\": \"olm-bundle-validation\",\n            \"log\": \"time=\\\"2021-10-25T15:59:43Z\\\" level=debug msg=\\\"Found manifests directory\\\" name=bundle-test\\ntime=\\\"2021-10-25T15:59:43Z\\\" level=debug msg=\\\"Found metadata directory\\\" name=bundle-test\\ntime=\\\"2021-10-25T15:59:43Z\\\" level=debug msg=\\\"Getting mediaType info from manifests directory\\\" name=bundle-test\\ntime=\\\"2021-10-25T15:59:43Z\\\" level=debug msg=\\\"Found annotations file\\\" name=bundle-test\\ntime=\\\"2021-10-25T15:59:43Z\\\" level=debug msg=\\\"Could not find optional dependencies file\\\" name=bundle-test\\n\",\n            \"state\": \"fail\",\n            \"errors\": [\n              \"Error: Value testpmd-operator-controller-manager: invalid service account found in bundle. sa name cannot match service account defined for deployment spec in CSV\"\n            ],\n            \"creationTimestamp\": null\n          }\n        ]\n      }\n    }\n  ]\n}\n"
time="2021-10-25T15:59:45Z" level=error msg="stderr: time=\"2021-10-25T15:59:37Z\" level=debug msg=\"Debug logging is set\"\n"
time="2021-10-25T15:59:45Z" level=info msg="check completed: ScorecardOlmSuiteCheck" ERROR="failed to run operator-sdk scorecard: exit status 1" result="failed to run operator-sdk scorecard: exit status 1"

Here is a corresponding part of results.json

    "errors": [
      {
        "name": "ScorecardOlmSuiteCheck",
        "elapsed_time": 8230,
        "description": "Operator-sdk scorecard OLM Test Suite Check",
        "help": "Check ScorecardOlmSuiteCheck encountered an error. Please review the artifacts/operator_bundle_scorecard_OlmSuiteCheck.json file for more information."
      },

Here are the errors I have when running operator-sdk scorecard directly

      "status": {
        "results": [
          {
            "name": "olm-bundle-validation",
            "log": "time=\"2021-10-25T15:57:55Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2021-10-25T15:57:55Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2021-10-25T15:57:55Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2021-10-25T15:57:55Z\" level=debug msg=\"Found annotations file\" name=bundle-test\ntime=\"2021-10-25T15:57:55Z\" level=debug msg=\"Could not find optional dependencies file\" name=bundle-test\n",
            "state": "fail",
            "errors": [
              "Error: Value testpmd-operator-controller-manager: invalid service account found in bundle. sa name cannot match service account defined for deployment spec in CSV"
            ],
            "creationTimestamp": null
          }
        ]
vikasmulaje commented 3 years ago

@tkrishtop this is a duplicate https://github.com/redhat-openshift-ecosystem/openshift-preflight/issues/169

tkrishtop commented 3 years ago

@vikasmulaje thank you. There is already a discussion in #169, so I close this one.

tkrishtop commented 2 years ago

Hello team, I'm testing the latest preflight tag b00d46ad3531b3059a2e1ffda3fa3531a669f0ec.

Here is the job that tests two operators: simple-demo-operator and testpmd-operator, here are all the log files.

  1. Good test case For the simple-demo-operator, the tests are passing, files operator_bundle_scorecard_BasicSpecCheck.json and operator_bundle_scorecard_OlmSuiteCheck.json are well-created and contain proper scorecard output.

Here is the exact version of the operator and its index image:

image: "quay.io/opdev/simple-demo-operator-bundle@sha256:eff7f86a54ef2a340dbf739ef955ab50397bef70f26147ed999e989cfc116b79"
index_image: "quay.io/opdev/simple-demo-operator-catalog:v0.0.3"
  1. Bad test case For the testpmd-operator, the tests are failing, files operator_bundle_scorecard_BasicSpecCheck.json and operator_bundle_scorecard_OlmSuiteCheck.json are now well-created (that was not the case for the 1.0.4 release). The problem is that they're empty.

Here is the exact version of the operator and its index image:

image: "quay.io/rh-nfv-int/testpmd-operator-bundle@sha256:5e28f883faacefa847104ebba1a1a22ee897b7576f0af6b8253c68b5c8f42815"
index_image: "quay.io/tkrishtop/index-testpmd-operator-bundle:v0.2.9"
tkrishtop commented 2 years ago

Tested it with preflight 1.0.5, the issue is fixed:

https://www.distributed-ci.io/jobs/cbfc5b94-654e-4a27-9494-a78051fa891b/files

    "failed": [
      {
        "name": "ValidateOperatorBundle",
        "elapsed_time": 114,
        "description": "Validating Bundle image that checks if it can validate the content and format of the operator bundle",
        "help": "Check ValidateOperatorBundle encountered an error. Please review the preflight.log file for more information.",
        "suggestion": "Valid bundles are defined by bundle spec, so make sure that this bundle conforms to that spec. More Information: https://github.com/operator-framework/operator-registry/blob/master/docs/design/operator-bundle.md",
        "knowledgebase_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide",
        "check_url": "https://connect.redhat.com/zones/containers/container-certification-policy-guide"
      },
      {
        "name": "ScorecardOlmSuiteCheck",
        "elapsed_time": 5309,
        "description": "Operator-sdk scorecard OLM Test Suite Check",
        "help": "Check ScorecardOlmSuiteCheck encountered an error. Please review the /artifacts/operator_bundle_scorecard_OlmSuiteCheck.json file for more information.",
        "suggestion": "See scorecard output for details, artifacts/operator_bundle_scorecard_OlmSuiteCheck.json",
        "knowledgebase_url": "https://sdk.operatorframework.io/docs/advanced-topics/scorecard/scorecard/#overview",
        "check_url": "https://sdk.operatorframework.io/docs/advanced-topics/scorecard/scorecard/#olm-test-suite"
      }
    ],

artifacts/operator_bundle_scorecard_OlmSuiteCheck.json is present and not empty:

{
  "kind": "TestList",
  "apiVersion": "scorecard.operatorframework.io/v1alpha3",
  "items": [
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test@sha256:3d80797ee809f09334e8af51a6aad94877de6ad790dffcab9122a853fe420050",
        "entrypoint": [
          "scorecard-test",
          "olm-bundle-validation"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-bundle-validation-test"
        },
        "storage": {
          "spec": {
            "mountPath": {}
          }
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-bundle-validation",
            "log": "time=\"2021-11-23T10:08:01Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2021-11-23T10:08:01Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2021-11-23T10:08:01Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2021-11-23T10:08:01Z\" level=debug msg=\"Found annotations file\" name=bundle-test\ntime=\"2021-11-23T10:08:01Z\" level=debug msg=\"Could not find optional dependencies file\" name=bundle-test\n",
            "state": "fail",
            "errors": [
              "Error: Value testpmd-operator-controller-manager: invalid service account found in bundle. sa name cannot match service account defined for deployment spec in CSV"
            ],
            "creationTimestamp": null
          }
        ]
      }
    }
  ]
}