search

submariner-io / submariner-operator

Operator that deploys the various Submariner components.
Apache License 2.0
105 stars 66 forks source link

Submariner Operator Significantly Outdated #301

Closed mrhillsman closed 4 years ago

mrhillsman commented 4 years ago

Hi,

I work on the Red Hat Operator Enablement team and I am starting this issue to push conversation forward regarding update of operator within OpenShift and Operatorhub.io

Currently the operator is failing to install using either of these options and one clear issue is the version of the operator listed in both is 0.0.1 but it appears 0.2.0 is now available.

https://github.com/operator-framework/community-operators/tree/master/upstream-community-operators/submariner https://github.com/operator-framework/community-operators/tree/master/community-operators/submariner

I am available to discuss how we can assist in getting this update done and ensuring future updates automagically appear within OpenShift and Operatorhub.io

Please feel free to reach out to me directly.

skitt commented 4 years ago

Hi,

Thanks for filing this issue, it just so happens that we (or rather, @Jaanki) are currently working on this!

We haven’t figured out why the current CSV no longer deploys (it did in the past). I have a WIP patch to update to 0.2.0, and it also fails to deploy, albeit differently (the install plan can’t even be built).

Any help would be most welcome.

mrhillsman commented 4 years ago

Thanks @skitt for getting back to me. I will pull down your WIP and see what I can do as well.

Jaanki commented 4 years ago

Error is: FAIL: Failed to reach AtLatestKnown subscription status for "submariner-sub" installplan status for "null": Error from server (NotFound): installplans.operators.coreos.com "null" not found Checking if subscriptions passes [ FAILED ]

Debugging further to find the root cause.

Jaanki commented 4 years ago

WIP patch export KUBECONFIG="$(kind get kubeconfig-path --name="operator-test")"

Process to reproduce:

  1. kind create cluster --name=operator-test
  2. make operator.test OP_PATH=community-operators/submariner/

Error (omitting other output)

Operator Deployment                               [  Processing  ]
    Applying object to cluster                    [  Processing  ]
    Applying object to cluster                    [  OK  ]
    Checking if subscriptions passes              [  Processing  ]
Checking subscription "submariner-sub" reaches "AtLatestKnown" state
subscription status for "submariner-sub": null
FAIL: Failed to reach AtLatestKnown subscription status for "submariner-sub"
installplan status for "null":
Error from server (NotFound): installplans.operators.coreos.com "null" not found
    Checking if subscriptions passes              [  FAILED  ]
make: *** [Makefile:85: operator.test] Error 1
make: *** [Makefile:56: operator.test] Error 2

Subscription yaml

$ kubectl describe subscription submariner-sub -n submariner-nutpf
Name:         submariner-sub
Namespace:    submariner-nutpf
Labels:       operator=test
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"operators.coreos.com/v1alpha1","kind":"Subscription","metadata":{"annotations":{},"labels":{"operator":"test"},"name":"subm...
API Version:  operators.coreos.com/v1alpha1
Kind:         Subscription
Metadata:
  Creation Timestamp:  2020-04-21T17:23:11Z
  Generation:          1
  Resource Version:    967
  Self Link:           /apis/operators.coreos.com/v1alpha1/namespaces/submariner-nutpf/subscriptions/submariner-sub
  UID:                 c6857d6a-83f4-11ea-ae9c-0242ac110002
Spec:
  Channel:           alpha
  Name:              submariner
  Source:            submariner-ocs
  Source Namespace:  submariner-nutpf
  Starting CSV:      submariner-operator.v0.2.0
Status:
  Catalog Health:
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              submariner-ocs
      Namespace:         submariner-nutpf
      Resource Version:  934
      UID:               c6a28cbd-83f4-11ea-ae9c-0242ac110002
    Healthy:             true
    Last Updated:        2020-04-21T17:23:12Z
  Conditions:
    Last Transition Time:  2020-04-21T17:23:12Z
    Message:               all available catalogsources are healthy
    Reason:                AllCatalogSourcesHealthy
    Status:                False
    Type:                  CatalogSourcesUnhealthy
  Last Updated:            2020-04-21T17:23:12Z
Events:                    <none>

OLM pods

$ kubectl get pods -n olm
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-66cf4f96f4-tprzm   1/1     Running   0          9m30s
olm-operator-9b64f8547-2ptvd        1/1     Running   0          9m30s
packageserver-6bb5c54b4c-4wb4t      1/1     Running   0          9m8s
packageserver-6bb5c54b4c-m7t56      1/1     Running   0          8m56s

Catalogoperator log

https://pastebin.com/dPnDXHQ4

olmoperator log

https://pastebin.com/rCpk30V2

Jaanki commented 4 years ago

catalogoperator.log olmoperator.log packageserver-6bb5c54b4c-4wb4t.log packageserver-6bb5c54b4c-m7t56.log

Also attaching files incase pastebins are not available

mrhillsman commented 4 years ago

Unfortunately the error is intermittent. I was able to copy over the newly created bundle files/folders to upstream-community-operators and it worked. I believe the script is looking to use this folder. However after working once it failed the second time. I moved to an OpenShift environment and see the following error:

CustomResourceDefinition.apiextensions.k8s.io "submariners.submariner.io" is invalid: [spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Schema:(apiextensions.CustomResourceValidation)(0xc010f6a198), Subresources:(apiextensions.CustomResourceSubresources)(0xc00d3f4c40), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Schema:(apiextensions.CustomResourceValidation)(0xc010f6a198), Subresources:(apiextensions.CustomResourceSubresources)(0xc00d3f4c40), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead)]

I believe this is related to a bug opened here: https://github.com/operator-framework/community-operators/pull/1272

mrhillsman commented 4 years ago

[openlab@node-0 community-operators]$ make operator.test OP_PATH=upstream-community-operators/submariner Pulling docker image [ Processing ] Pulling docker image [ OK ] KIND cluster auto-creation [ selected ] Start KIND [ Processing ] xStart KIND [ OK ] Building catalog image [ Processing ] Building catalog image [ OK ] xOperator version detected [ 0.2.0 ] Creating namespace [ Processing ] Creating namespace [ OK ] Lint operator metadata [ Processing ] Lint operator metadata [ OK ] Install OLM [ Processing ] Install OLM [ OK ] Building manifests [ Processing ] Building manifests [ OK ] Operator Deployment [ Processing ] Applying object to cluster [ Processing ] Applying object to cluster [ OK ] Checking if subscriptions passes [ Processing ] Checking if subscriptions passes [ OK ] Checking if CSV passes [ Processing ] Check that clusterserviceversion "submariner-operator.v0.2.0" status is set to "Succeeded" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" clusterserviceversion phase for "submariner-operator.v0.2.0": "Pending" failed to reach Succeeded clusterserviceversion status for "submariner-operator.v0.2.0" map[conditions:[map[lastTransitionTime:2020-04-23T00:51:05Z lastUpdateTime:2020-04-23T00:51:05Z message:requirements not yet checked phase:Pending reason:RequirementsUnknown] map[lastTransitionTime:2020-04-23T00:51:05Z lastUpdateTime:2020-04-23T00:51:05Z message:one or more requirements couldn't be found phase:Pending reason:RequirementsNotMet]] lastTransitionTime:2020-04-23T00:51:05Z lastUpdateTime:2020-04-23T00:51:05Z message:one or more requirements couldn't be found phase:Pending reason:RequirementsNotMet requirementStatus:[map[group:operators.coreos.com kind:ClusterServiceVersion message:CSV minKubeVersion (1.11.0) less than server version (v1.17.0) name:submariner-operator.v0.2.0 status:Present version:v1alpha1] map[group:apiextensions.k8s.io kind:CustomResourceDefinition message:CRD is present and Established condition is true name:clusters.submariner.io status:Present uuid:16e88a9e-fcaf-4062-891a-1635782ec34c version:v1beta1] map[group:apiextensions.k8s.io kind:CustomResourceDefinition message:CRD is present and Established condition is true name:endpoints.submariner.io status:Present uuid:de00f2c8-8467-4d64-99b9-c67a5776f88c version:v1beta1] map[group:apiextensions.k8s.io kind:CustomResourceDefinition message:CRD is not present name:submariners.submariner.io status:NotPresent version:v1beta1] map[group: kind:ServiceAccount message:Service account does not exist name:submariner-operator status:NotPresent version:v1]]]Pod submariner-operator listed in CSV was not found in namespace submariner-zdogh Checking if CSV passes [ FAILED ] make: [Makefile:85: operator.test] Error 1 make: [Makefile:56: operator.test] Error 2

Jaanki commented 4 years ago

Operator is being updated to v0.2.0 in https://github.com/operator-framework/community-operators/pull/1677

nyechiel commented 4 years ago

https://github.com/operator-framework/community-operators/pull/1696

SteveMattar commented 4 years ago

up

mrhillsman commented 4 years ago

CRD for submariner submariner CR - https://gist.github.com/mrhillsman/18d6831bf36caafeaeac001bf0813ec1 - specific to deprecation of CRD spec (apiextensions.k8s.io/v1beta1).

CRD for submariner servicediscoveries CR - https://gist.github.com/mrhillsman/224158a40ab6a11d7e28e0aec59e4573 - specific to deprecation of CRD spec (apiextensions.k8s.io/v1beta1).

SteveMattar commented 4 years ago

A plan for updating submariner-operator

1) package the operator (use operator-sdk v0.19.1)

2) Test the Operator before submitting a PR make operator.test OP_PATH=upstream-community-operators/submariner

3) preview operator in operatorhub.io preview tool

4) create 2 separated PRs in operator-framework/community-operators by following this checklist include the update operator package in each directory:

5) update our upstream CI to automate the process. These scripts may help

6) same for downstream CI (CPaaS)

SteveMattar commented 4 years ago

we might need to migrate our submariner-operator project from a legacy layout to the new Kubebuilder layout. there are some migration guides on the official website.

SteveMattar commented 4 years ago

new PR for v0.5.0 https://github.com/operator-framework/community-operators/pull/2132

SteveMattar commented 4 years ago

https://github.com/operator-framework/community-operators/pull/2132 was merged