search

emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.36k stars 683 forks source link

Unable to create `Host` resource due to tls cert error from emissary-apiext #5728

Open fs185143 opened 2 months ago

fs185143 commented 2 months ago

Describe the bug Cannot create Host resource after upgrading emissary-apiext and emissary-ingress

To Reproduce Steps to reproduce the behavior:

  1. Create Host resource on an environment running ambassador 2.0.4
  2. Upgrade from 2.0.4 to 3.9.1
  3. Observe emissary-apiext start as expected
  4. Try create Host resource again
  5. Observe error in emissary-apiext logs: 
    Host/emissary/ingress-host dry-run failed, error: conversion webhook for getambassador.io/v3alpha1, Kind=Host failed: Post "https://emissary-apiext.emissary-system.svc:443/?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
  6. emissary/emissary-ingress deployment fails indefinitely as it includes this Host
  7. Restart emissary-apiext pod in emissary-system namespace
  8. Host applies fine and emissary-ingress deployment can proceed

Expected behavior A clear and concise description of what you expected to happen.

Should be able to apply Host without getting above webhook error from emissary-system/emissary-apiext

Versions (please complete the following information):

Additional context Wondering if some sort of race condition is occurring

fs185143 commented 2 months ago

Am I correct in thinking that the certificate in question is the value of caBundle here?

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  generation: 2
  labels:
    app.kubernetes.io/instance: emissary-apiext
    app.kubernetes.io/managed-by: kubectl_apply_-f_emissary-apiext.yaml
    app.kubernetes.io/name: emissary-apiext
    app.kubernetes.io/part-of: emissary-apiext
  name: hosts.getambassador.io
  resourceVersion: "244433"
  uid: ef8bf370-0ca4-485c-859d-2a083a67db40
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        caBundle: LS0t...0tLQo=
fs185143 commented 2 months ago

something i noticed after running the b64 decoded caBundle value through openssl x509 is that the validity is

        Validity
            Not Before: Jul 25 07:40:24 2024 GMT
            Not After : Jul 25 07:40:24 2025 GMT

whereas the CRD's status shows

status:
  acceptedNames:
    categories:
    - ambassador-crds
    kind: Host
    listKind: HostList
    plural: hosts
    singular: host
  conditions:
  - lastTransitionTime: "2024-07-25T12:09:21Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2024-07-25T12:09:21Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v2

and metadata.creationTimestamp of creationTimestamp: "2024-07-25T12:09:21Z"

the cert error from above was at 07:42:19.561

fs185143 commented 2 months ago

suspect these logs from emissary-system/emissary-apiext may be relevant

time="2024-07-25 11:57:06.6676" level=info msg="Configuring conversion for \"authservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.6901" level=info msg="Configuring conversion for \"consulresolvers.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.6995" level=info msg="Configuring conversion for \"devportals.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.7135" level=info msg="Configuring conversion for \"hosts.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.7348" level=info msg="Configuring conversion for \"kubernetesendpointresolvers.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.7446" level=info msg="Configuring conversion for \"kubernetesserviceresolvers.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:07.0713" level=info msg="Configuring conversion for \"logservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:07.4710" level=info msg="Configuring conversion for \"mappings.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:07.8720" level=info msg="Configuring conversion for \"modules.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:08.2707" level=info msg="Configuring conversion for \"ratelimitservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:08.6711" level=info msg="Configuring conversion for \"tcpmappings.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.0709" level=info msg="Configuring conversion for \"tlscontexts.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.4713" level=info msg="Configuring conversion for \"tracingservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.8716" level=error msg="goroutine \"/configure-crds\" exited with error: 13 errors:\n 1. customresourcedefinitions.apiextensions.k8s.io \"authservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 2. customresourcedefinitions.apiextensions.k8s.io \"consulresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 3. customresourcedefinitions.apiextensions.k8s.io \"devportals.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 4. customresourcedefinitions.apiextensions.k8s.io \"hosts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 5. customresourcedefinitions.apiextensions.k8s.io \"kubernetesendpointresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 6. customresourcedefinitions.apiextensions.k8s.io \"kubernetesserviceresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 7. customresourcedefinitions.apiextensions.k8s.io \"logservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 8. customresourcedefinitions.apiextensions.k8s.io \"mappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 9. customresourcedefinitions.apiextensions.k8s.io \"modules.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 10. customresourcedefinitions.apiextensions.k8s.io \"ratelimitservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 11. customresourcedefinitions.apiextensions.k8s.io \"tcpmappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 12. customresourcedefinitions.apiextensions.k8s.io \"tlscontexts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 13. customresourcedefinitions.apiextensions.k8s.io \"tracingservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.8717" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=apiext PID=1 THREAD=":shutdown_logger"
time="2024-07-25 11:57:09.8720" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8720" level=info msg="    /configure-crds: exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8721" level=info msg="    /serve-http    : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8721" level=info msg="    /serve-https   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8722" level=error msg="shut down with error error: 13 errors:\n 1. customresourcedefinitions.apiextensions.k8s.io \"authservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 2. customresourcedefinitions.apiextensions.k8s.io \"consulresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 3. customresourcedefinitions.apiextensions.k8s.io \"devportals.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 4. customresourcedefinitions.apiextensions.k8s.io \"hosts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 5. customresourcedefinitions.apiextensions.k8s.io \"kubernetesendpointresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 6. customresourcedefinitions.apiextensions.k8s.io \"kubernetesserviceresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 7. customresourcedefinitions.apiextensions.k8s.io \"logservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 8. customresourcedefinitions.apiextensions.k8s.io \"mappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 9. customresourcedefinitions.apiextensions.k8s.io \"modules.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 10. customresourcedefinitions.apiextensions.k8s.io \"ratelimitservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 11. customresourcedefinitions.apiextensions.k8s.io \"tcpmappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 12. customresourcedefinitions.apiextensions.k8s.io \"tlscontexts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 13. customresourcedefinitions.apiextensions.k8s.io \"tracingservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=apiext PID=1
fs185143 commented 2 months ago

seems related to https://github.com/emissary-ingress/emissary/pull/5468 which afaik is not included in v3.9.1

cindymullins-dw commented 2 months ago

Yes, at a glance it does seem that fix could be related and was merged after the v.3.9.1 release. Future releases of Emissary will depend on the maintainers collectively so that schedule is currently undecided. It's possible to build Emissary from source to capture PRs that have been merged but not yet released in a new version, if you wanted to test that.

fs185143 commented 2 months ago

Yes, at a glance it does seem that fix could be related and was merged after the v.3.9.1 release. Future releases of Emissary will depend on the maintainers collectively so that schedule is currently undecided. It's possible to build Emissary from source to capture PRs that have been merged but not yet released in a new version, if you wanted to test that.

Is there a discussion or anything that relates to recent news regarding the future of emissary releases? We would only be able to target stable releases/patch fixes

fs185143 commented 2 months ago

also after investigating patches, i think this is likely to be the fix for our particular issue https://github.com/emissary-ingress/emissary/commit/c8edb1626480f6657f79fe80aa0a8df74f986a0f - once it gets released anyway

kflynn commented 2 months ago

@fs185143 We're looking into an Emissary 3.10 that will include that fix -- watch this space. 🙂