search

operator-framework / operator-sdk

SDK for building Kubernetes applications. Provides high level APIs, useful abstractions, and project scaffolding.
https://sdk.operatorframework.io
Apache License 2.0
7.25k stars 1.75k forks source link

Memcached sample is not passing in the operator-sdk scorecard bundle test #4564

Closed camilamacedo86 closed 2 years ago

camilamacedo86 commented 3 years ago

Bug Report

Memcached sample is not passing in the operator-sdk scorecard bundle test.

What did you do?

$ cd testdata/go/v3/memcached-operator
$ kind create cluster
$ operator-sdk scorecard bundle --output=json

What did you expect to see?

What did you see instead? Under which circumstances?

See the errors faced:

            "name": "olm-status-descriptors",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "memcacheds.cache.example.com does not have a status descriptor"

            "name": "olm-crds-have-resources",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\n",
            "state": "fail",
            "errors": [
              "Owned CRDs do not have resources specified"

Following the full result: 

$ operator-sdk scorecard bundle --output=json 
{
  "kind": "TestList",
  "apiVersion": "scorecard.operatorframework.io/v1alpha3",
  "items": [
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-crds-have-resources"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-crds-have-resources-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-crds-have-resources",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\n",
            "state": "fail",
            "errors": [
              "Owned CRDs do not have resources specified"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-status-descriptors"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-status-descriptors-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-status-descriptors",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "memcacheds.cache.example.com does not have a status descriptor"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "basic-check-spec"
        ],
        "labels": {
          "suite": "basic",
          "test": "basic-check-spec-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "basic-check-spec",
            "state": "pass"
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-bundle-validation"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-bundle-validation-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-bundle-validation",
            "log": "time=\"2021-02-22T11:17:19Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=info msg=\"Found annotations file\" name=bundle-test\ntime=\"2021-02-22T11:17:19Z\" level=info msg=\"Could not find optional dependencies file\" name=bundle-test\n",
            "state": "pass"
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-spec-descriptors"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-spec-descriptors-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-spec-descriptors",
            "log": "Loaded ClusterServiceVersion: memcached-operator.v0.0.1\nLoaded 1 Custom Resources from alm-examples\n",
            "state": "fail",
            "errors": [
              "size does not have a spec descriptor"
            ],
            "suggestions": [
              "Add a spec descriptor for size"
            ]
          }
        ]
      }
    },
    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test:v1.4.0",
        "entrypoint": [
          "scorecard-test",
          "olm-crds-have-validation"
        ],
        "labels": {
          "suite": "olm",
          "test": "olm-crds-have-validation-test"
        }
      },
      "status": {
        "results": [
          {
            "name": "olm-crds-have-validation",
            "log": "Loaded 1 Custom Resources from alm-examples\nLoaded CustomresourceDefinitions: [\u0026CustomResourceDefinition{ObjectMeta:{memcacheds.cache.example.com      0 0001-01-01 00:00:00 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[] map[controller-gen.kubebuilder.io/version:v0.4.1] [] []  []},Spec:CustomResourceDefinitionSpec{Group:cache.example.com,Names:CustomResourceDefinitionNames{Plural:memcacheds,Singular:memcached,ShortNames:[],Kind:Memcached,ListKind:MemcachedList,Categories:[],},Scope:Namespaced,Versions:[]CustomResourceDefinitionVersion{CustomResourceDefinitionVersion{Name:v1alpha1,Served:true,Storage:true,Schema:\u0026CustomResourceValidation{OpenAPIV3Schema:\u0026JSONSchemaProps{ID:,Schema:,Ref:nil,Description:Memcached is the Schema for the memcacheds API,Type:object,Format:,Title:,Default:nil,Maximum:nil,ExclusiveMaximum:false,Minimum:nil,ExclusiveMinimum:false,MaxLength:nil,MinLength:nil,Pattern:,MaxItems:nil,MinItems:nil,UniqueItems:false,MultipleOf:nil,Enum:[]JSON{},MaxProperties:nil,MinProperties:nil,Required:[],Items:nil,AllOf:[]JSONSchemaProps{},OneOf:[]JSONSchemaProps{},AnyOf:[]JSONSchemaProps{},Not:nil,Properties:map[string]JSONSchemaProps{apiVersion: {  \u003cnil\u003e APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},kind: {  \u003cnil\u003e Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},metadata: {  \u003cnil\u003e  object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},spec: {  \u003cnil\u003e MemcachedSpec defines the desired state of Memcached object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[foo:{  \u003cnil\u003e Foo is an example field of Memcached. Edit memcached_types.go to remove/update string   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e} size:{  \u003cnil\u003e Size defines the number of Memcached instances integer int32  nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e}] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},status: {  \u003cnil\u003e MemcachedStatus defines the observed state of Memcached object   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] nil [] [] [] nil map[nodes:{  \u003cnil\u003e Nodes store the name of the pods which are running Memcached instances array   nil \u003cnil\u003e false \u003cnil\u003e false \u003cnil\u003e \u003cnil\u003e  \u003cnil\u003e \u003cnil\u003e false \u003cnil\u003e [] \u003cnil\u003e \u003cnil\u003e [] \u0026JSONSchemaPropsOrArray{Schema:\u0026JSONSchemaProps{ID:,Schema:,Ref:nil,Description:,Type:string,Format:,Title:,Default:nil,Maximum:nil,ExclusiveMaximum:false,Minimum:nil,ExclusiveMinimum:false,MaxLength:nil,MinLength:nil,Pattern:,MaxItems:nil,MinItems:nil,UniqueItems:false,MultipleOf:nil,Enum:[]JSON{},MaxProperties:nil,MinProperties:nil,Required:[],Items:nil,AllOf:[]JSONSchemaProps{},OneOf:[]JSONSchemaProps{},AnyOf:[]JSONSchemaProps{},Not:nil,Properties:map[string]JSONSchemaProps{},AdditionalProperties:nil,PatternProperties:map[string]JSONSchemaProps{},Dependencies:JSONSchemaDependencies{},AdditionalItems:nil,Definitions:JSONSchemaDefinitions{},ExternalDocs:nil,Example:nil,Nullable:false,XPreserveUnknownFields:nil,XEmbeddedResource:false,XIntOrString:false,XListMapKeys:[],XListType:nil,XMapType:nil,},JSONSchemas:[]JSONSchemaProps{},} [] [] [] nil map[] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e}] nil map[] map[] nil map[] nil nil false \u003cnil\u003e false false [] \u003cnil\u003e \u003cnil\u003e},},AdditionalProperties:nil,PatternProperties:map[string]JSONSchemaProps{},Dependencies:JSONSchemaDependencies{},AdditionalItems:nil,Definitions:JSONSchemaDefinitions{},ExternalDocs:nil,Example:nil,Nullable:false,XPreserveUnknownFields:nil,XEmbeddedResource:false,XIntOrString:false,XListMapKeys:[],XListType:nil,XMapType:nil,},},Subresources:\u0026CustomResourceSubresources{Status:\u0026CustomResourceSubresourceStatus{},Scale:nil,},AdditionalPrinterColumns:[]CustomResourceColumnDefinition{},Deprecated:false,DeprecationWarning:nil,},},Conversion:nil,PreserveUnknownFields:false,},Status:CustomResourceDefinitionStatus{Conditions:[]CustomResourceDefinitionCondition{},AcceptedNames:CustomResourceDefinitionNames{Plural:,Singular:,ShortNames:[],Kind:,ListKind:,Categories:[],},StoredVersions:[],},}]\n",
            "state": "pass"
          }
        ]
      }
    }
  ]
}

Enviroment:

estroz commented 3 years ago

These tests can be made to pass by CSV markers added by the sample generator to Go operator samples. Another way to make changes to make these pass needs implementing for Ansible/Helm samples, ex. kustomize patch files on the CSV.

camilamacedo86 commented 3 years ago

I am removing myself as assigned from this one and adding help wanted and good first issue since follows what required to get done.

To achieve that we need to improve the mocks in https://github.com/operator-framework/operator-sdk/tree/master/hack/generate/samples to add what is required to pass in the scorecard for these test and then change the tests in https://github.com/operator-framework/operator-sdk/blob/9fa13f31e2b265253c105315ee12cc439e7b4174/test/common/scorecard.go#L70-L73, All should pass in this case.

Also, please feel free to ping me if you need help to work on that.

openshift-bot commented 3 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot commented 3 years ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot commented 2 years ago

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci[bot] commented 2 years ago

@openshift-bot: Closing this issue.

In response to [this](https://github.com/operator-framework/operator-sdk/issues/4564#issuecomment-973704613): >Rotten issues close after 30d of inactivity. > >Reopen the issue by commenting `/reopen`. >Mark the issue as fresh by commenting `/remove-lifecycle rotten`. >Exclude this issue from closing again by commenting `/lifecycle frozen`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
camilamacedo86 commented 2 years ago

/lifecycle frozen

ima1mai commented 2 years ago

I could reproduce the error locally. I'll try looking into it.