search

usrbinkat / pulumi-kongee-on-k8s

GNU Affero General Public License v3.0
0 stars 0 forks source link

BUG: ingress objects failing to associate to existing services #1

Open usrbinkat opened 2 years ago

usrbinkat commented 2 years ago

Final deploy seems to function but pulumi errors on association failure with an error code 255 

Previewing update (kphd)

View Live: https://app.pulumi.com/usrbinkat/kong/kphd/previews/d0c182a6-9f6f-4315-9581-036a9e0f0b29

    pulumi:pulumi:Stack kong-kphd running 
    kubernetes:helm.sh/v3:Chart controlplane  
    kubernetes:helm.sh/v3:Chart dataplane  
    kubernetes:core/v1:Service kong/dataplane-kong-proxy  
    kubernetes:core/v1:ConfigMap kong/dataplane-kong-custom-dbless-config  
    kubernetes:apps/v1:Deployment kong/dataplane-kong  
    kubernetes:core/v1:ServiceAccount kong/controlplane-kong  
    kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding controlplane-kong  
    kubernetes:rbac.authorization.k8s.io/v1:RoleBinding kong/controlplane-kong  
    kubernetes:core/v1:Service kong/controlplane-kong-clustertelemetry  
    kubernetes:core/v1:Service kong/controlplane-kong-cluster  
    kubernetes:core/v1:Service kong/controlplane-kong-portal  
    kubernetes:core/v1:Service kong/controlplane-kong-admin  
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal update warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin update warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:rbac.authorization.k8s.io/v1:Role kong/controlplane-kong  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongplugins.configuration.konghq.com  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongclusterplugins.configuration.konghq.com  
    kubernetes:core/v1:Service kong/controlplane-kong-portalapi  
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager update warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi update warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition udpingresses.configuration.konghq.com  
    kubernetes:core/v1:Service kong/controlplane-kong-manager  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongconsumers.configuration.konghq.com  
    kubernetes:rbac.authorization.k8s.io/v1:ClusterRole controlplane-kong  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition tcpingresses.configuration.konghq.com  
    kubernetes:batch/v1:Job kong/controlplane-kong-init-migrations  
    kubernetes:batch/v1:Job kong/controlplane-kong-post-upgrade-migrations  
    kubernetes:batch/v1:Job kong/controlplane-kong-pre-upgrade-migrations  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongingresses.configuration.konghq.com  
    kubernetes:apps/v1:Deployment kong/controlplane-kong  
    pulumi:pulumi:Stack kong-kphd  

Diagnostics:
  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-manager):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.

  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-portalapi):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.

  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-admin):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.

  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-portal):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.

Updating (kphd)

View Live: https://app.pulumi.com/usrbinkat/kong/kphd/updates/27

    pulumi:pulumi:Stack kong-kphd running 
    kubernetes:helm.sh/v3:Chart controlplane  
    kubernetes:helm.sh/v3:Chart dataplane  
    kubernetes:core/v1:ConfigMap kong/dataplane-kong-custom-dbless-config  
    kubernetes:core/v1:Service kong/dataplane-kong-proxy  
    kubernetes:apps/v1:Deployment kong/dataplane-kong  
    kubernetes:rbac.authorization.k8s.io/v1:RoleBinding kong/controlplane-kong  
    kubernetes:core/v1:Service kong/controlplane-kong-portal  
    kubernetes:core/v1:ServiceAccount kong/controlplane-kong  
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin updating warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi updating warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:core/v1:Service kong/controlplane-kong-cluster  
    kubernetes:core/v1:Service kong/controlplane-kong-admin  
    kubernetes:core/v1:Service kong/controlplane-kong-clustertelemetry  
    kubernetes:rbac.authorization.k8s.io/v1:Role kong/controlplane-kong  
    kubernetes:core/v1:Service kong/controlplane-kong-portalapi  
    kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding controlplane-kong  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongconsumers.configuration.konghq.com  
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal updating warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager  warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager updating warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongclusterplugins.configuration.konghq.com  
    kubernetes:core/v1:Service kong/controlplane-kong-manager  
    kubernetes:rbac.authorization.k8s.io/v1:ClusterRole controlplane-kong  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongplugins.configuration.konghq.com  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition udpingresses.configuration.konghq.com  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition tcpingresses.configuration.konghq.com  
    kubernetes:batch/v1:Job kong/controlplane-kong-init-migrations  
    kubernetes:batch/v1:Job kong/controlplane-kong-post-upgrade-migrations  
    kubernetes:batch/v1:Job kong/controlplane-kong-pre-upgrade-migrations  
    kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kongingresses.configuration.konghq.com  
    kubernetes:apps/v1:Deployment kong/controlplane-kong  
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin updating [1/3] Finding a matching service for each Ingress path
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin updating No matching service found for ingress rule: "manager.kong.home.arpa/api" -> "controlplane-kong-admin"
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin updating No matching service found for ingress rule: "manager.kong.home.arpa/api" -> ""
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi updating [1/3] Finding a matching service for each Ingress path
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi updating No matching service found for ingress rule: "portal.kong.home.arpa/api" -> "controlplane-kong-portalapi"
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi updating No matching service found for ingress rule: "portal.kong.home.arpa/api" -> ""
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal updating [1/3] Finding a matching service for each Ingress path
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal updating No matching service found for ingress rule: "portal.kong.home.arpa/" -> "controlplane-kong-portal"
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal updating No matching service found for ingress rule: "portal.kong.home.arpa/" -> ""
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager updating [1/3] Finding a matching service for each Ingress path
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager updating No matching service found for ingress rule: "manager.kong.home.arpa/" -> "controlplane-kong-manager"
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager updating [2/3] Waiting for update of .status.loadBalancer with hostname/IP
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager updating No matching service found for ingress rule: "manager.kong.home.arpa/" -> ""
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin updating error: 2 errors occurred:
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-admin **updating failed** error: 2 errors occurred:
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi updating error: 2 errors occurred:
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portalapi **updating failed** error: 2 errors occurred:
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal updating error: 2 errors occurred:
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-portal **updating failed** error: 2 errors occurred:
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager updating error: 2 errors occurred:
 ~  kubernetes:extensions/v1beta1:Ingress kong/controlplane-kong-manager **updating failed** error: 2 errors occurred:
    pulumi:pulumi:Stack kong-kphd running error: update failed
    pulumi:pulumi:Stack kong-kphd **failed** 1 error

Diagnostics:
  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-admin):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.
    error: 2 errors occurred:
        * the Kubernetes API server reported that "kong/controlplane-kong-admin" failed to fully initialize or become live: 'controlplane-kong-admin' timed out waiting to be Ready
        * Ingress has at least one rule that does not target any Service. Field '.spec.rules[].http.paths[].backend.serviceName' may not match any active Service

  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-portal):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.
    error: 2 errors occurred:
        * the Kubernetes API server reported that "kong/controlplane-kong-portal" failed to fully initialize or become live: 'controlplane-kong-portal' timed out waiting to be Ready
        * Ingress has at least one rule that does not target any Service. Field '.spec.rules[].http.paths[].backend.serviceName' may not match any active Service

  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-portalapi):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.
    error: 2 errors occurred:
        * the Kubernetes API server reported that "kong/controlplane-kong-portalapi" failed to fully initialize or become live: 'controlplane-kong-portalapi' timed out waiting to be Ready
        * Ingress has at least one rule that does not target any Service. Field '.spec.rules[].http.paths[].backend.serviceName' may not match any active Service

  kubernetes:extensions/v1beta1:Ingress (kong/controlplane-kong-manager):
    warning: apiVersion "extensions/v1beta1/Ingress" was removed in Kubernetes 1.20. Use "networking.k8s.io/v1beta1/Ingress" instead.
    See https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.20.md#deprecation for more information.
    error: 2 errors occurred:
        * the Kubernetes API server reported that "kong/controlplane-kong-manager" failed to fully initialize or become live: 'controlplane-kong-manager' timed out waiting to be Ready
        * Ingress has at least one rule that does not target any Service. Field '.spec.rules[].http.paths[].backend.serviceName' may not match any active Service

  pulumi:pulumi:Stack (kong-kphd):
    error: update failed

Resources:
    27 unchanged

Duration: 10m4s

Actual state of services and ingresses:

NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
postgres-postgresql-headless         ClusterIP      None             <none>         5432/TCP                     6h1m
postgres-postgresql                  ClusterIP      10.152.183.154   <none>         5432/TCP                     6h1m
dataplane-kong-proxy                 LoadBalancer   10.152.183.2     192.168.1.31   80:32486/TCP,443:31793/TCP   72m
controlplane-kong-clustertelemetry   ClusterIP      10.152.183.24    <none>         8006/TCP                     72m
controlplane-kong-portalapi          ClusterIP      10.152.183.96    <none>         8447/TCP                     72m
controlplane-kong-cluster            ClusterIP      10.152.183.110   <none>         8005/TCP                     72m
controlplane-kong-manager            ClusterIP      10.152.183.224   <none>         8445/TCP                     72m
controlplane-kong-portal             ClusterIP      10.152.183.188   <none>         8446/TCP                     72m
controlplane-kong-admin              ClusterIP      10.152.183.174   <none>         8444/TCP                     72m

NAME                          CLASS    HOSTS                    ADDRESS        PORTS     AGE
controlplane-kong-portal      <none>   portal.kong.home.arpa    192.168.1.31   80, 443   72m
controlplane-kong-manager     <none>   manager.kong.home.arpa   192.168.1.31   80, 443   72m
controlplane-kong-portalapi   <none>   portal.kong.home.arpa    192.168.1.31   80, 443   72m
controlplane-kong-admin       <none>   manager.kong.home.arpa   192.168.1.31   80, 443   72m

Kong Manager Service and Ingress yaml:

apiVersion: v1
kind: Service
metadata:
  annotations:
    konghq.com/protocol: https
  labels:
    app.kubernetes.io/instance: controlplane
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: "2.5"
    helm.sh/chart: kong-2.3.0
  name: controlplane-kong-manager
  namespace: kong
spec:
  clusterIP: 10.152.183.224
  clusterIPs:
  - 10.152.183.224
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: kong-manager-tls
    port: 8445
    protocol: TCP
    targetPort: 8445
  selector:
    app.kubernetes.io/component: app
    app.kubernetes.io/instance: controlplane
    app.kubernetes.io/name: kong
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/https-redirect-status-code: "301"
    konghq.com/protocols: https
    konghq.com/strip-path: "true"
    kubernetes.io/ingress.class: kong
  generation: 1
  labels:
    app.kubernetes.io/instance: controlplane
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: "2.5"
    helm.sh/chart: kong-2.3.0
  name: controlplane-kong-manager
  namespace: kong
spec:
  rules:
  - host: manager.kong.home.arpa
    http:
      paths:
      - backend:
          service:
            name: controlplane-kong-manager
            port:
              number: 8445
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - manager.kong.home.arpa
    secretName: kong-tls
status:
  loadBalancer:
    ingress:
    - ip: 192.168.1.31
usrbinkat commented 2 years ago

Addtl Info:

~$ k get -nkong po
NAME                                 ENDPOINTS                             AGE
postgres-postgresql-headless         10.1.128.216:5432                     22h
postgres-postgresql                  10.1.128.216:5432                     22h
dataplane-kong-proxy                 10.1.128.226:8080,10.1.128.226:8443   17h
controlplane-kong-cluster            10.1.128.223:8005                     17h
controlplane-kong-clustertelemetry   10.1.128.223:8006                     17h
controlplane-kong-portalapi          10.1.128.223:8447                     17h
controlplane-kong-manager            10.1.128.223:8445                     17h
controlplane-kong-portal             10.1.128.223:8446                     17h
controlplane-kong-admin              10.1.128.223:8444                     17h

~$ k get -nkong ep
NAME                                 READY   STATUS    RESTARTS   AGE
postgres-postgresql-0                1/1     Running   0          22h
dataplane-kong-55b7675999-cwbfk      1/1     Running   0          17h
controlplane-kong-65f957d5c9-bcfxp   2/2     Running   0          17h
usrbinkat commented 2 years ago

Pulumi up with debug enabled pulumi--debug.log

usrbinkat commented 2 years ago

Relevant source:

https://github.com/pulumi/pulumi-kubernetes/blob/876eaea150e4c50f8b53f3345bd060139780626e/provider/pkg/await/ingress.go#L406

https://github.com/pulumi/pulumi-kubernetes/blob/876eaea150e4c50f8b53f3345bd060139780626e/provider/pkg/await/ingress.go#L322