[bitnami/external-dns] External-DNS Chart Broken with 7.3.3 due to removal of protected CRD group annotation

[pinkfloydx33]

Name and Version

bitnami/external-dns

What architecture are you using?

None

What steps will reproduce the bug?

Attempt to upgrade or install the external-dns chart at version 7.3.3. In our case, an automated upgrade via Flux has started spamming alerts across all of our environments where minor/patch upgrades are automatically performed.

What is the expected behavior?

Helm chart upgrades/installs

What do you see instead?

Installation fails with:

Helm upgrade failed: cannot patch "dnsendpoints.externaldns.k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "dnsendpoints.externaldns.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111

Additional information

This is because the automated upgrade has removed the protected annotation on the CRDs:

• Commit @ Line12
• PR

This annotation is required on CRDs if the group is k8s.io, kubernetes.io, or ends with .k8s.io, .kubernetes.io which is applicable here because the CRD group is apiextensions.k8s.io.

This change needs to be reverted or else the chart is unusable.

[carrodher]

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

[pinkfloydx33]

@carrodher I wouldn't know where to begin as the original change was made by some form of automation, which tells me that it too would need to be fixed. I also unfortunately need approval from my company before contributing, even to tools we use, and that would likely take more time than others broken by this change would appreciate.

[andeke07]

I have added the annotation to the pull request on the original external-dns project, I believe this is where the Bitnami automation gets its information from so if this gets approved hopefully the next release will contain it again?

[andeke07]

This has been fixed now with https://github.com/kubernetes-sigs/external-dns/commit/f46676fdbbe336cc3c86f68d5c96c7cf920aeba0

I don't know the next steps though, I suppose a release needs to be done and then the bitnami chart updated to refer to the new release?

[rouke-broersma]

I think the release of such updates is automated, not sure how they are triggered but I expect that it will come within a reasonable time.

[carrodher]

Our automation looks for new releases at https://github.com/kubernetes-incubator/external-dns. When a new release is cut there, our automated test & release process is triggered. As part of that process, the upstream CRDs are compared and the ones in the Bitnami chart are updated to match the upstream, i.e https://github.com/bitnami/charts/pull/25962/commits/375ee3ba1adfabacb8c8ccb29fb97b78c15e92c1

[herrbpl]

Still broken in 7.5.0

[andeke07]

external-dns hasn't issued a new release yet so there's nothing for the new chart to go off of

[raiomarco]

So... there's a fix? a workaround? if not, what version of the chart should i use? i tried the 7.5.2 but it didn't work :(

[cheinema]

@raiomarco Since the problem seems to have occurred starting with chart version 7.3.3, v7.3.2 should be the last stable version for now. We are still waiting for a new release in https://github.com/kubernetes-sigs/external-dns to include the fix.

[MaxAnderson95]

@cheinema Unless I'm missing something, the v0.14.2 release appears to have the fix. Is there anything else that needs to be done before merging the fix into the chart?

[rouke-broersma]

@cheinema Unless I'm missing something, the v0.14.2 release appears to have the fix. Is there anything else that needs to be done before merging the fix into the chart?

You're looking at the external dns helm chart managed by external dns. This is not the source of crds for the bitnami chart.

[MaxAnderson95]

@rouke-broersma I knew I was missing something! Thanks.

[Atoms]

so seems crd cannot be updated manually, as there is ci pipeline which allows only bitnami bot to update crd. and there is no release from external-dns side which would include api-approved annotation.

[pinkfloydx33]

This chart is effectively broken for now... Does anyone have a way to workaround it or are we SOL for now?

[rouke-broersma]

This chart is effectively broken for now... Does anyone have a way to workaround it or are we SOL for now?

I think you could simply deploy the required resources yourself (updated crd, clusterRole on crd, clusterRoleBinding to serviceaccount from chart): https://github.com/search?q=repo%3Abitnami%2Fcharts%20path%3A%2F%5Ebitnami%5C%2Fexternal-dns%5C%2Ftemplates%5C%2F%2F%20.Values.crd.create&type=code

That should be sufficient until upstream releases a new version.

[pinkfloydx33]

Ok thanks. We use Flux for management, I'm sure theres a way to do that, just haven't looked into it yet. Hopefully the upstream fixes it soon...

[hawkesn]

Still broken with 7.5.7

[andeke07]

@hawkesn the chart relies on external-dns coming out with another release which they haven't done in a month: https://github.com/kubernetes-sigs/external-dns/releases

The next time external-dns release, it will have the fix, which will then get embedded in this chart.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[MaxAnderson95]

Commenting to keep this issue open

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[MaxAnderson95]

Commenting to keep this issue open ...again

[venkatamutyala]

Commenting to keep this active and opened an issue here requesting a release be cut: https://github.com/kubernetes-sigs/external-dns/issues/4657

[mpechner-akasa]

This is real.

[Mykyta-Serbynevskyi]

Hi there, This issue also impacts our services, as of now it is not possible to upgrade the external-dns helmchart from 7.2.1 to latest 8.3.4:

>  helm upgrade -i external-dns bitnami/external-dns -n clusterutils -f values-external-dns.yaml --version 8.3.4
Error: UPGRADE FAILED: cannot patch "dnsendpoints.externaldns.k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "dnsendpoints.externaldns.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111

Would appreciate if a workaround is issued for this asap. Thank you.

[ruifung]

Currently I think, the only workaround I've found (as a consumer of the chart) is to use a post-renderer to patch the annotation back in.

This is an excerpt from my HelmRelease definition (FluxCD) for this that patches the required annotation back in when it's deployed.

  postRenderers:
    - kustomize:
        patches:
          - patch: |
              - op: "add"
                path: "/metadata/annotations/api-approved.kubernetes.io"
                value: "https://github.com/kubernetes-sigs/external-dns/pull/2007"
            target:
              kind: CustomResourceDefinition
              name: dnsendpoints.externaldns.k8s.io

[mpechner-akasa]

Ignore this, I had to back port to 7.3.2. external-dns is broken. Would be interested in the tests and "works on my system" comments from the maintainers. I am on eks, k8s vers 1.24.

For me this was an upgrade from 7.5.5 - which had an issue with the CRD. The latest, 8.3.4.

I removed the CRD and applied the new one manually. I removed the helm chart setting to update CRD. External DNS is not bad since there is just one CRD to apply.

All good now.

The helm chart maintainer should handle this. But for me since other charts require I manually maintain the CRD, not an issue. What's one more.

[pinkfloydx33]

@ruifung thanks for that. I had been thinking about patching with Flux but wasn't sure how it would handle patching CRDs. It does the trick though!

[mpechner-akasa]

Seems to be fixed here: https://github.com/kubernetes-sigs/external-dns/pull/4488 I guess wait until bitnami updated teir chart, or we all just move to the external-dns chart.

[venkatamutyala]

Seems to be fixed here: kubernetes-sigs/external-dns#4488 I guess wait until bitnami updated teir chart, or we all just move to the external-dns chart.

I think external-dns needs to still release the fix and then bitnami will pick it up automatically. The PR you shared was merged on May 20th but the last release was May 16th.

[mpechner-akasa]

Seems to be fixed here: kubernetes-sigs/external-dns#4488 I guess wait until bitnami updated teir chart, or we all just move to the external-dns chart.

I think external-dns needs to still release the fix and then bitnami will pick it up automatically. The PR you shared was merged on May 20th but the last release was May 16th.

Then bitnami is broken, has been broken, and should not have updated to external-dns 0.14.2.

[rouke-broersma]

The crds are an addon feature that is disabled by default and is most likely not widely used in production scenarios. It's not required for the core functionality of external dns. It's annoying that it hasn't been fixed yet but it's hardly the end of the world. The external dns chart with it's support for specific providers out of the box is imo way more convenient than the upstream chart.

[venkatamutyala]

The crds are an addon feature that is disabled by default and is most likely not widely used in production scenarios. It's not required for the core functionality of external dns. It's annoying that it hasn't been fixed yet but it's hardly the end of the world. The external dns chart with it's support for specific providers out of the box is imo way more convenient than the upstream chart.

I can't speak for everyone but I'm using this resource across 10+ production clusters.

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint

I also see others on github have been toggling the crd.create=true as well:

https://github.com/search?q=external-dns+crd.create%3Dtrue&type=code

I think this needs to get fixed properly. I just don't know how given the lack of traction in getting a release cut here:

https://github.com/kubernetes-sigs/external-dns/issues/4657#issuecomment-2285149866

[javsalgar]

Let me check with the team. Maybe, for the time being, we can disable the automation for this specific asset and re-enable it once they cut the release.

[fmulero]

This problem comes from kubernets-sig/external-dns There are several issues about this:

• https://github.com/kubernetes-sigs/external-dns/issues/4487
• https://github.com/kubernetes-sigs/external-dns/issues/4483

The issue was fixed this PR but there is no new releases with that change.

From the bitnami side, we update automatically the CRDs and the containers based on new releases. We shouldn't use main branch for the CRDs as source of truth because this and the containers could be not properly aligned. Until there's an upstream release, you can use kustomize (or similar tools) to apply the changes in the Bitnami chart, sth like:

helm template external-dns oci://registry-1.docker.io/bitnamicharts/external-dns --set crd.create=true (...) | kubectl apply -k (...)

[rouke-broersma]

This problem comes from kubernets-sig/external-dns There are several issues about this:

• api-approved.kubernetes.io missing from CRD since 0.14.2 kubernetes-sigs/external-dns#4487
• DNSEndpoint CRD has had its approval annotation removed kubernetes-sigs/external-dns#4483

The issue was fixed this PR but there is no new releases with that change.

From the bitnami side, we update automatically the CRDs and the containers based on new releases. We shouldn't use main branch for the CRDs as source of truth because this and the containers could be not properly aligned. Until there's an upstream release, you can use kustomize (or similar tools) to apply the changes in the Bitnami chart, sth like:

helm template external-dns oci://registry-1.docker.io/bitnamicharts/external-dns --set crd.create=true (...) | kubectl apply -k (...)

I don't have this capability in my environment, I can only perform helm deployments. This does not solve my problem. I agree that you should not rely on main for the crds, however the bitnami chart crds have been broken for months now.

Would it instead be possible to temporarily use the latest helm chart release as the source of the crd instead? The helm chart has been updated with the fix to the crd.

[juan131]

There's a PR attempt to address this, see https://github.com/bitnami/charts/pull/27434

[venkatamutyala]

Looks like we may have a fix. Anyone roll out >= 8.3.5 in production yet? :)

[rouke-broersma]

I updated without issues

[javsalgar]

Thanks for letting us know! Can we close this issue then?

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[venkatamutyala]

@javsalgar I think we need to revert the change once the upstream chart has been released?

[javsalgar]

Correct, when upstream releases, we can re-enable it.

[venkatamutyala]

@javsalgar it is time. :) A new release was dropped yesterday and appears to have the fix:

https://github.com/kubernetes-sigs/external-dns/releases/tag/v0.15.0

fix: re-add api-approved.kubernetes.io annotation by @morremeyer in https://github.com/kubernetes-sigs/external-dns/pull/4488

[javsalgar]

Thanks for letting us know! I created a PR ⏫

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[venkatamutyala]

Commenting so this issue doesn't fall stale. I think we are just waiting on https://github.com/bitnami/charts/pull/29266 to get merged.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[venkatamutyala]

Still a work in progress. Hasn't been merged it: https://github.com/bitnami/charts/pull/29266