search

weaveworks / flux2-openshift

OperatorHub submission repo for Flux2
Apache License 2.0
4 stars 4 forks source link

Updating Flux 0.36 -> 0.37 gives error: CannotUpdate #17

Open WillNilges opened 2 years ago

WillNilges commented 2 years ago

A few days ago, Flux started trying to update itself from 0.36 to 0.37. It then began failing. The 0.37 version is stuck in a Pending state due to a RequirementsNotMet: one or more requirements couldn't be found error. 0.36 was in a Failing state.

image

OLM Logs:

I1128 02:12:22.717658       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, subscription flux exists, subscription flux requires @existing/flux-system//flux.v0.37.0, @existing/flux-system//flux.v0.37.0 and @existing/flux-system//flux.v0.36.0 provide HelmRelease (helm.toolkit.fluxcd.io/v2beta1)
time="2022-11-28T02:12:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:23.716878       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: @existing/flux-system//flux.v0.37.0 and @existing/flux-system//flux.v0.36.0 originate from package flux, clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, subscription flux exists, subscription flux requires @existing/flux-system//flux.v0.37.0
time="2022-11-28T02:12:24Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:24.717529       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: subscription flux requires @existing/flux-system//flux.v0.37.0, @existing/flux-system//flux.v0.37.0 and @existing/flux-system//flux.v0.36.0 provide Kustomization (kustomize.toolkit.fluxcd.io/v1beta2), clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, subscription flux exists
time="2022-11-28T02:12:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:25.716471       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: subscription flux exists, clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, @existing/flux-system//flux.v0.37.0 and @existing/flux-system//flux.v0.36.0 provide Bucket (source.toolkit.fluxcd.io/v1beta1), subscription flux requires @existing/flux-system//flux.v0.37.0
time="2022-11-28T02:12:26Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:26.717348       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: @existing/flux-system//flux.v0.37.0 and @existing/flux-system//flux.v0.36.0 provide ImagePolicy (image.toolkit.fluxcd.io/v1beta1), clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, subscription flux exists, subscription flux requires @existing/flux-system//flux.v0.37.0
time="2022-11-28T02:12:27Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:27.718395       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: subscription flux exists, @existing/flux-system//flux.v0.36.0 and @existing/flux-system//flux.v0.37.0 originate from package flux, clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, subscription flux requires @existing/flux-system//flux.v0.37.0
time="2022-11-28T02:12:28Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:28.718722       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: subscription flux exists, clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, @existing/flux-system//flux.v0.36.0 and @existing/flux-system//flux.v0.37.0 provide ImageRepository (image.toolkit.fluxcd.io/v1beta1), subscription flux requires @existing/flux-system//flux.v0.37.0
time="2022-11-28T02:12:29Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:29.718087       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: subscription flux exists, clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, @existing/flux-system//flux.v0.36.0 and @existing/flux-system//flux.v0.37.0 provide Provider (notification.toolkit.fluxcd.io/v1beta1), subscription flux requires @existing/flux-system//flux.v0.37.0
time="2022-11-28T02:12:30Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:30.717538       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: subscription flux requires @existing/flux-system//flux.v0.37.0, subscription flux exists, @existing/flux-system//flux.v0.37.0 and @existing/flux-system//flux.v0.36.0 provide HelmChart (source.toolkit.fluxcd.io/v1beta2), clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription
time="2022-11-28T02:12:31Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
I1128 02:12:31.717771       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"flux-system", UID:"203c94e7-1df1-4a48-a2e2-d882dd13c68d", APIVersion:"v1", ResourceVersion:"539695358", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' constraints not satisfiable: subscription flux exists, subscription flux requires @existing/flux-system//flux.v0.37.0, clusterserviceversion flux.v0.36.0 exists and is not referenced by a subscription, @existing/flux-system//flux.v0.37.0 and @existing/flux-system//flux.v0.36.0 provide HelmChart (source.toolkit.fluxcd.io/v1beta1)

I attempted to perform the steps in this RedHat article, but they didn't work (Delete the Subscription and CSVs, re-install).

Logs post re-install

time="2022-11-28T02:24:22Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2022-11-28T02:24:22Z" level=info msg=syncing id=b/zWa ip=install-k8vlf namespace=flux-system phase=Installing
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=ocirepository.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=Service" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=webhook-receiver.service.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=ClusterRole" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=crd-controller-flux-system.clusterrole.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=kustomization.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=ClusterRoleBinding" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=cluster-reconciler-flux-system.clusterrolebinding.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=receiver.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=provider.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=Service" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=source-controller.service.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=helmrelease.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=imageupdateautomation.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=imagepolicy.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=gitrepository.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=helmrepository.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=helmchart.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=imagerepository.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=bucket.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=flux.v0.37.0.clusterserviceversion.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=alert.crd.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=ClusterRoleBinding" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=crd-controller-flux-system.clusterrolebinding.yaml
time="2022-11-28T02:24:23Z" level=info msg="added to bundle, Kind=Service" configmap=openshift-marketplace/117acbb8c3636cc0e69f7d4db4f352bb74df05c0154698c1603646f69a7a6d8 key=notification-controller.service.yaml
time="2022-11-28T02:24:23Z" level=error msg="risk of data loss updating \"imagepolicies.image.toolkit.fluxcd.io\": new CRD removes version v1alpha1 that is listed as a stored version on the existing CRD"

Anyone seen this issue before? I must admit, I'm not all that familiar with how operators work, so any help or advice would be appreciated.

WillNilges commented 2 years ago

Upon attempting to re-install from the OperatorHub, I get this error on the Subscription page

risk of data loss updating "imagepolicies.image.toolkit.fluxcd.io": new CRD removes version v1alpha1 that is listed as a stored version on the existing CRD
WillNilges commented 2 years ago

Uninstalled Flux via FluxCLI with flux uninstall and re-installed via the OperatorHub. That seems to have worked :)

kingdonb commented 2 years ago

Thanks @WillNilges for helping us trace down this issue!

The upgrade should be seamless, I'm not sure how we can address this issue in a future release to prevent this error, or if it's possible to bypass that error somehow else without uninstalling. (Maybe we can add upgrade testing to our validation plan?)

I'm making a lot of assumptions, but I believe that this is caused by the final deprecation of the v1alpha APIs that came in Flux v0.37, and that it won't happen again. The v1alpha APIs required a manual translation into v1beta in a much earlier version of Flux, such that there was no translator requirement to ensure that v1alpha APIs are automatically upgraded into beta when the new API version comes out.

But beta APIs have a higher guarantee level for production readiness in Kubernetes and in Flux. So the deprecation of alpha APIs is a signal that the General Availability release is coming soon 🎉 which means I think this will not happen again.

I think a manual upgrade of the CRDs might have also worked around this issue. I don't think OLM is being too conservative here, I think that it's probably unusual for operators that are still based on prerelease APIs to land in the catalog ahead of the API stability guarantees being hardened. I would be speculating about a lot of that stuff though, just giving some color to the report that hopefully helps explain a bit how we got here, on the way to Flux GA.