errordeveloper
commented
3 years ago Closed errordeveloper closed 3 years ago
errordeveloper
commented
3 years ago errordeveloper
commented
3 years ago Scorecard test were include amongst the generate configs, should be quite easy to add. I omitted these cause I didn't know what these are for and what is actually being tested.
errordeveloper
commented
3 years ago It seems to fail with something more meaningful now:
===== Test: operator-scorecard-tests =====
The operator FAILED the scorecard criteria of passing all basic tests.
========== Output of null-null-scorecard-results.json =============
{
"kind": "ScorecardOutput",
"apiVersion": "osdk.openshift.io/v1alpha2",
"metadata": {
"creationTimestamp": null
},
"log": "time=\"2021-02-17T19:30:12Z\" level=info msg=\"Using config file: /home/jenkins/agent/workspace/cvp-isv-operator-metadata-validation-test/osdk-scorecard.yml\"\ntime=\"2021-02-17T19:30:14Z\" level=error msg=\"Plugin `Basic Tests` failed with error (could not get GVKs for resource(s) in file: /home/jenkins/agent/scorecard-cr-files/first.cr.yaml, due to error: (failed to unmarshal object spec: Object 'Kind' is missing in '{\\\"metadata\\\":{\\\"namespace\\\":\\\"test-operator\\\"}}'))\"\ntime=\"2021-02-17T19:30:16Z\" level=error msg=\"Plugin `OLM Integration` failed with error (could not get GVKs for resource(s) in file: /home/jenkins/agent/scorecard-cr-files/first.cr.yaml, due to error: (failed to unmarshal object spec: Object 'Kind' is missing in '{\\\"metadata\\\":{\\\"namespace\\\":\\\"test-operator\\\"}}'))\"\n",
"results": []
}
======== Debug output of the operator container =========
2021-02-17T19:28:30.489Z INFO cmd Version {"Go Version": "go1.15.5", "GOOS": "linux", "GOARCH": "amd64", "helm-operator": "v1.3.0", "commit": "1abf57985b43bf6a59dcd18147b3c574fa57d3f6"}
2021-02-17T19:28:30.490Z INFO cmd Watching single namespace. {"Namespace": "cilium"}
I0217 19:28:31.541228 1 request.go:645] Throttling request took 1.036534677s, request: GET:http://eyJhcGlWZXJzaW9uIjoiIiwia2luZCI6IiIsIm5hbWUiOiJzY29yZWNhcmQiLCJ1aWQiOiIiLCJOYW1lc3BhY2UiOiJ0ZXN0LW9wZXJhdG9yIn0K@localhost:8889/apis/flowcontrol.apiserver.k8s.io/v1alpha1?timeout=32s
2021-02-17T19:28:32.695Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2021-02-17T19:28:32.696Z INFO helm.controller Watching resource {"apiVersion": "cilium.io/v1alpha1", "kind": "CiliumConfig", "namespace": "cilium", "reconcilePeriod": "1m0s"}
I0217 19:28:32.696366 1 leaderelection.go:243] attempting to acquire leader lease test-operator/cilium-olm...
2021-02-17T19:28:32.696Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I0217 19:28:32.710005 1 leaderelection.go:253] successfully acquired lease test-operator/cilium-olm
2021-02-17T19:28:32.710Z INFO controller-runtime.manager.controller.ciliumconfig-controller Starting EventSource {"source": "kind source: cilium.io/v1alpha1, Kind=CiliumConfig"}
2021-02-17T19:28:32.710Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"test-operator","name":"cilium-olm","uid":"142ad590-e1c9-4f66-b3ab-8d9fe949a236","apiVersion":"v1","resourceVersion":"24973"}, "reason": "LeaderElection", "message": "ip-10-0-162-152_2ca23a8c-e399-40a1-9373-0f1315659ac2 became leader"}
2021-02-17T19:28:32.811Z INFO controller-runtime.manager.controller.ciliumconfig-controller Starting Controller
2021-02-17T19:28:32.811Z INFO controller-runtime.manager.controller.ciliumconfig-controller Starting workers {"worker count": 2}
======== Error output of the operator scorecard test =========
Output of null-scorecard-errors.txt:
time="2021-02-17T19:30:12Z" level=debug msg="Debug logging is set"
-------------------
Execution Reference:
-> /cvp/cvp-isv-operator-metadata-validation-test/certified-ospid-e31ac831-7e72-42bb-baf9-f392ef7ea622-72de8bd8-e4a8-418f-8660-d90d2cc9c691/a39de67d-0eb2-41fc-89ed-01b945f49446/
Also, local tests say this:
$ operator-sdk scorecard bundles/cilium.v1.9.3
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.0.1
Entrypoint: [scorecard-test olm-spec-descriptors]
Labels:
"suite":"olm"
"test":"olm-spec-descriptors-test"
Results:
Name: olm-spec-descriptors
State: pass
Log:
Loaded ClusterServiceVersion: cilium.v1.9.3
Loaded 0 Custom Resources from alm-examples
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.0.1
Entrypoint: [scorecard-test basic-check-spec]
Labels:
"suite":"basic"
"test":"basic-check-spec-test"
Results:
Name: basic-check-spec
State: pass
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.0.1
Entrypoint: [scorecard-test olm-bundle-validation]
Labels:
"suite":"olm"
"test":"olm-bundle-validation-test"
Results:
Name: olm-bundle-validation
State: pass
Suggestions:
Warning: Value : (cilium.v1.9.3) example annotations not found
Log:
time="2021-02-17T20:49:14Z" level=debug msg="Found manifests directory" name=bundle-test
time="2021-02-17T20:49:14Z" level=debug msg="Found metadata directory" name=bundle-test
time="2021-02-17T20:49:14Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test
time="2021-02-17T20:49:14Z" level=info msg="Found annotations file" name=bundle-test
time="2021-02-17T20:49:14Z" level=info msg="Could not find optional dependencies file" name=bundle-test
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.0.1
Entrypoint: [scorecard-test olm-status-descriptors]
Labels:
"suite":"olm"
"test":"olm-status-descriptors-test"
Results:
Name: olm-status-descriptors
State: pass
Log:
Loaded ClusterServiceVersion: cilium.v1.9.3
Loaded 0 Custom Resources from alm-examples
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.0.1
Entrypoint: [scorecard-test olm-crds-have-validation]
Labels:
"suite":"olm"
"test":"olm-crds-have-validation-test"
Results:
Name: olm-crds-have-validation
State: pass
Log:
Loaded 0 Custom Resources from alm-examples
Loaded CustomresourceDefinitions: [&CustomResourceDefinition{ObjectMeta:{ciliumconfigs.cilium.io 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:CustomResourceDefinitionSpec{Group:cilium.io,Names:CustomResourceDefinitionNames{Plural:ciliumconfigs,Singular:ciliumconfig,ShortNames:[],Kind:CiliumConfig,ListKind:CiliumList,Categories:[],},Scope:Namespaced,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1alpha1,Served:true,Storage:true,Schema:nil,Subresources:&CustomResourceSubresources{Status:&CustomResourceSubresourceStatus{},Scale:nil,},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},},},Conversion:nil,PreserveUnknownFields:false,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}]
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.0.1
Entrypoint: [scorecard-test olm-crds-have-resources]
Labels:
"suite":"olm"
"test":"olm-crds-have-resources-test"
Results:
Name: olm-crds-have-resources
State: fail
Errors:
Owned CRDs do not have resources specified
Log:
Loaded ClusterServiceVersion: cilium.v1.9.3
$... level=error msg=\"Plugin `Basic Tests` failed with error (could not get GVKs for resource(s) in file: /home/jenkins/agent/scorecard-cr-files/first.cr.yaml, due to error: (failed to unmarshal object spec: Object 'Kind' is missing in '{\\\"metadata\\\":{\\\"namespace\\\":\\\"test-operator\\\"}}'))\"\ntime=\"2021-02-17T19:30:16Z\" level=error msg=\"Plugin `OLM Integration` failed with error (could not get GVKs for resource(s) in file: /home/jenkins/agent/scorecard-cr-files/first.cr.yaml, due to error: (failed to unmarshal object spec: Object 'Kind' is missing in '{\\\"metadata\\\":{\\\"namespace\\\":\\\"test-operator\\\"}}'))\"\n",From a conversation with support folks, this error is due to missing example CR.
errordeveloper
commented
3 years ago New scan failure:
===== Test: operator-scorecard-tests =====
The operator FAILED the scorecard criteria of passing all basic tests.
========== Output of CiliumConfig-cilium-openshift-default-scorecard-results.json =============
{
"kind": "ScorecardOutput",
"apiVersion": "osdk.openshift.io/v1alpha2",
"metadata": {
"creationTimestamp": null
},
"log": "time=\"2021-02-18T12:24:11Z\" level=info msg=\"Using config file: /home/jenkins/agent/workspace/cvp-isv-operator-metadata-validation-test/osdk-scorecard.yml\"\n",
"results": [
{
"name": "Writing into CRs has an effect",
"description": "A CR sends PUT/POST requests to the API server to modify resources in response to spec block changes",
"labels": {
"necessity": "required",
"suite": "basic",
"test": "writingintocrshaseffecttest"
},
"state": "pass"
},
{
"name": "Spec Block Exists",
"description": "Custom Resource has a Spec Block",
"labels": {
"necessity": "required",
"suite": "basic",
"test": "checkspectest"
},
"state": "pass"
},
{
"name": "Status Block Exists",
"description": "Custom Resource has a Status Block",
"labels": {
"necessity": "required",
"suite": "basic",
"test": "checkstatustest"
},
"state": "fail",
"suggestions": [
"Add a 'status' field to your Custom Resource"
]
},
{
"name": "Bundle Validation Test",
"description": "Validates bundle contents",
"labels": {
"necessity": "required",
"suite": "olm",
"test": "bundlevalidationtest"
},
"state": "fail",
"errors": [
"unable to find the OLM 'bundle' directory which is required for this test"
]
},
{
"name": "Provided APIs have validation",
"description": "All CRDs have an OpenAPI validation subsection",
"labels": {
"necessity": "required",
"suite": "olm",
"test": "crdshavevalidationtest"
},
"state": "fail",
"suggestions": [
"Add CRD validation for CiliumConfig/v1alpha1"
]
},
{
"name": "Owned CRDs have resources listed",
"description": "All Owned CRDs contain a resources subsection",
"labels": {
"necessity": "required",
"suite": "olm",
"test": "crdshaveresourcestest"
},
"state": "fail",
"suggestions": [
"If it would be helpful to an end-user to understand or troubleshoot your CR, consider adding resources [configmaps/v1 events/v1 ciliumconfigs/v1alpha1] to the resources section for owned CRD CiliumConfig"
]
},
{
"name": "Spec fields with descriptors",
"description": "All spec fields have matching descriptors in the CSV",
"labels": {
"necessity": "required",
"suite": "olm",
"test": "specdescriptorstest"
},
"state": "fail",
"suggestions": [
"Add a spec descriptor for cni",
"Add a spec descriptor for endpointRoutes",
"Add a spec descriptor for hubble",
"Add a spec descriptor for ipam",
"Add a spec descriptor for nativeRoutingCIDR",
"Add a spec descriptor for prometheus"
]
},
{
"name": "Status fields with descriptors",
"description": "All status fields have matching descriptors in the CSV",
"labels": {
"necessity": "required",
"suite": "olm",
"test": "statusdescriptorstest"
},
"state": "pass"
}
]
}
======== Debug output of the operator container =========
2021-02-18T12:20:41.049Z INFO cmd Version {"Go Version": "go1.15.5", "GOOS": "linux", "GOARCH": "amd64", "helm-operator": "v1.3.0", "commit": "1abf57985b43bf6a59dcd18147b3c574fa57d3f6"}
2021-02-18T12:20:41.051Z INFO cmd Watching single namespace. {"Namespace": "cilium"}
I0218 12:20:42.101389 1 request.go:645] Throttling request took 1.038376545s, request: GET:http://eyJhcGlWZXJzaW9uIjoiIiwia2luZCI6IiIsIm5hbWUiOiJzY29yZWNhcmQiLCJ1aWQiOiIiLCJOYW1lc3BhY2UiOiJ0ZXN0LW9wZXJhdG9yIn0K@localhost:8889/apis/migration.k8s.io/v1alpha1?timeout=32s
2021-02-18T12:20:43.255Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2021-02-18T12:20:43.256Z INFO helm.controller Watching resource {"apiVersion": "cilium.io/v1alpha1", "kind": "CiliumConfig", "namespace": "cilium", "reconcilePeriod": "1m0s"}
I0218 12:20:43.257043 1 leaderelection.go:243] attempting to acquire leader lease test-operator/cilium-olm...
2021-02-18T12:20:43.257Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I0218 12:20:43.275675 1 leaderelection.go:253] successfully acquired lease test-operator/cilium-olm
2021-02-18T12:20:43.275Z INFO controller-runtime.manager.controller.ciliumconfig-controller Starting EventSource {"source": "kind source: cilium.io/v1alpha1, Kind=CiliumConfig"}
2021-02-18T12:20:43.276Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"test-operator","name":"cilium-olm","uid":"2691a0f5-7ed7-46b8-9890-5207dff3df75","apiVersion":"v1","resourceVersion":"24936"}, "reason": "LeaderElection", "message": "ip-10-0-128-26_6a3bfe88-7fb2-4914-8d5b-71b4ffca7009 became leader"}
2021-02-18T12:20:43.376Z INFO controller-runtime.manager.controller.ciliumconfig-controller Starting Controller
2021-02-18T12:20:43.376Z INFO controller-runtime.manager.controller.ciliumconfig-controller Starting workers {"worker count": 2}
======== Error output of the operator scorecard test =========
Output of cilium-openshift-default-scorecard-errors.txt:
time="2021-02-18T12:24:11Z" level=debug msg="Debug logging is set"
-------------------
Execution Reference:
-> /cvp/cvp-isv-operator-metadata-validation-test/certified-ospid-e31ac831-7e72-42bb-baf9-f392ef7ea622-89881ae9-4f0e-4bb4-b901-4deed199a16f/2989b266-16fb-4650-b506-acbf2036311f/
Locally only one test is failing now:
$ operator-sdk scorecard bundles/cilium.v1.9.3
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-crds-have-validation]
Labels:
"suite":"olm"
"test":"olm-crds-have-validation-test"
Results:
Name: olm-crds-have-validation
State: pass
Suggestions:
Add CRD validation for CiliumConfig/v1alpha1
Log:
Loaded 1 Custom Resources from alm-examples
Loaded CustomresourceDefinitions: [&CustomResourceDefinition{ObjectMeta:{ciliumconfigs.cilium.io 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:CustomResourceDefinitionSpec{Group:cilium.io,Names:CustomResourceDefinitionNames{Plural:ciliumconfigs,Singular:ciliumconfig,ShortNames:[],Kind:CiliumConfig,ListKind:CiliumList,Categories:[],},Scope:Namespaced,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1alpha1,Served:true,Storage:true,Schema:nil,Subresources:&CustomResourceSubresources{Status:&CustomResourceSubresourceStatus{},Scale:nil,},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},Deprecated:false,DeprecationWarning:nil,},},Conversion:nil,PreserveUnknownFields:false,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}]
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-status-descriptors]
Labels:
"suite":"olm"
"test":"olm-status-descriptors-test"
Results:
Name: olm-status-descriptors
State: pass
Log:
Loaded ClusterServiceVersion: cilium.v1.9.3
Loaded 1 Custom Resources from alm-examples
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-bundle-validation]
Labels:
"suite":"olm"
"test":"olm-bundle-validation-test"
Results:
Name: olm-bundle-validation
State: pass
Log:
time="2021-02-18T17:10:18Z" level=debug msg="Found manifests directory" name=bundle-test
time="2021-02-18T17:10:18Z" level=debug msg="Found metadata directory" name=bundle-test
time="2021-02-18T17:10:18Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test
time="2021-02-18T17:10:18Z" level=info msg="Found annotations file" name=bundle-test
time="2021-02-18T17:10:18Z" level=info msg="Could not find optional dependencies file" name=bundle-test
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-spec-descriptors]
Labels:
"suite":"olm"
"test":"olm-spec-descriptors-test"
Results:
Name: olm-spec-descriptors
State: fail
Suggestions:
Add a spec descriptor for cni
Add a spec descriptor for ipam
Add a spec descriptor for prometheus
Add a spec descriptor for hubble
Add a spec descriptor for nativeRoutingCIDR
Add a spec descriptor for endpointRoutes
Errors:
cni does not have a spec descriptor
ipam does not have a spec descriptor
prometheus does not have a spec descriptor
hubble does not have a spec descriptor
nativeRoutingCIDR does not have a spec descriptor
endpointRoutes does not have a spec descriptor
Log:
Loaded ClusterServiceVersion: cilium.v1.9.3
Loaded 1 Custom Resources from alm-examples
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test basic-check-spec]
Labels:
"suite":"basic"
"test":"basic-check-spec-test"
Results:
Name: basic-check-spec
State: pass
--------------------------------------------------------------------------------
Image: quay.io/operator-framework/scorecard-test:v1.3.0
Entrypoint: [scorecard-test olm-crds-have-resources]
Labels:
"suite":"olm"
"test":"olm-crds-have-resources-test"
Results:
Name: olm-crds-have-resources
State: pass
Log:
Loaded ClusterServiceVersion: cilium.v1.9.3
$```