new semver test failure in scorecard

errordeveloper commented 3 years ago

This has failed for 1.9.7:

===== Test: operator-scorecard-tests =====

 The operator FAILED the scorecard criteria of passing all basic tests.

========== Output of CiliumConfig-cilium-openshift-default-scorecard-results.json =============

{
  "kind": "ScorecardOutput",
  "apiVersion": "osdk.openshift.io/v1alpha2",
  "metadata": {
    "creationTimestamp": null
  },
  "log": "time=\"2021-05-18T08:51:56Z\" level=info msg=\"Using config file: /home/jenkins/agent/workspace/cvp-isv-operator-metadata-validation-test/osdk-scorecard.yml\"\ntime=\"2021-05-18T08:51:56Z\" level=error msg=\"Plugin `Basic Tests` failed with error (error validating ClusterServiceVersion: Error: Value : (cilium.v1.9.7-0236092) metadata.name \\\"cilium.v1.9.7-0236092\\\" contains an invalid semver \\\"v1.9.7-0236092\\\"\\n)\"\ntime=\"2021-05-18T08:51:56Z\" level=error msg=\"Plugin `OLM Integration` failed with error (error validating ClusterServiceVersion: Error: Value : (cilium.v1.9.7-0236092) metadata.name \\\"cilium.v1.9.7-0236092\\\" contains an invalid semver \\\"v1.9.7-0236092\\\"\\n)\"\n",
  "results": []
}

======== Debug output of the operator container =========

2021-05-18T08:50:31.221Z    INFO    cmd Version {"Go Version": "go1.15.10", "GOOS": "linux", "GOARCH": "amd64", "helm-operator": "v1.4.0+git", "commit": "98f30d59ade2d911a7a8c76f0169a7de0dec37a0"}
2021-05-18T08:50:31.222Z    INFO    cmd Watching single namespace.  {"Namespace": "test-operator"}
I0518 08:50:32.272953       1 request.go:655] Throttling request took 1.039339673s, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1?timeout=32s
2021-05-18T08:50:33.432Z    INFO    controller-runtime.metrics  metrics server is starting to listen    {"addr": ":8080"}
2021-05-18T08:50:33.433Z    INFO    controller-runtime.injectors-warning    Injectors are deprecated, and will be removed in v0.10.x
2021-05-18T08:50:33.433Z    INFO    controller-runtime.injectors-warning    Injectors are deprecated, and will be removed in v0.10.x
2021-05-18T08:50:33.433Z    INFO    helm.controller Watching resource   {"apiVersion": "cilium.io/v1alpha1", "kind": "CiliumConfig", "namespace": "test-operator", "reconcilePeriod": "1m0s"}
I0518 08:50:33.434003       1 leaderelection.go:243] attempting to acquire leader lease test-operator/cilium-olm...
2021-05-18T08:50:33.434Z    INFO    controller-runtime.manager  starting metrics server {"path": "/metrics"}
I0518 08:50:33.450720       1 leaderelection.go:253] successfully acquired lease test-operator/cilium-olm
2021-05-18T08:50:33.450Z    INFO    controller-runtime.manager.controller.ciliumconfig-controller   Starting EventSource    {"source": "kind source: cilium.io/v1alpha1, Kind=CiliumConfig"}
2021-05-18T08:50:33.451Z    DEBUG   controller-runtime.manager.events   Normal  {"object": {"kind":"ConfigMap","namespace":"test-operator","name":"cilium-olm","uid":"89e4fd00-f19a-4813-af9b-a8de02f17618","apiVersion":"v1","resourceVersion":"24180"}, "reason": "LeaderElection", "message": "ip-10-0-154-165_e8481734-fd99-4be8-bbb7-043f1d6d6e69 became leader"}
2021-05-18T08:50:33.551Z    INFO    controller-runtime.manager.controller.ciliumconfig-controller   Starting Controller
2021-05-18T08:50:33.551Z    INFO    controller-runtime.manager.controller.ciliumconfig-controller   Starting workers    {"worker count": 2} 

======== Error output of the operator scorecard test =========

 Output of cilium-openshift-default-scorecard-errors.txt: 

time="2021-05-18T08:51:56Z" level=debug msg="Debug logging is set"

-------------------
Execution Reference:

-> /cvp/cvp-isv-operator-metadata-validation-test/certified-ospid-e31ac831-7e72-42bb-baf9-f392ef7ea622-28c062c4-fb0b-4cd2-9f53-fe3df9d9ec90/4bca9f15-3231-4efa-be44-4943214f0639/

errordeveloper commented 3 years ago

So the error could be returned from here:

Note Operator SDK 1.4.0 is used at present, which use API package v0.5.3.

But this doesn't look quite right, as the formatting would differ....

errordeveloper commented 3 years ago

Ah, looks like Dockerfile linked above is not representative and even older version of the SDK is used.

This validation check has actually been removed: https://github.com/operator-framework/api/commit/ac177123d749ce2128ff4258d101c898d936dad2

errordeveloper commented 3 years ago

From a quick test, it looks like prepending a any non-numeric character should pass.

errordeveloper commented 3 years ago

It's possible that this test always been there, the issue is to do with the leading zero...

cilium / cilium-olm

new semver test failure in scorecard #46