Comprehensive Reference/Documentation

flyer299 commented 4 years ago

Is there a place that I am not seeing that will give me comprehensive documentation & reference for the different types of Ingress Options that are available with AGIC in the Ingress Yaml file? There are several use cases I want to start exploring, but I feel like a dummy for not even knowing where to start.

They include:

Having multiple websites running. Example: sampleapp-a.mydomain.com and sampleapp-b.mydomain.com.
URL based routing. sampleapp.mydomain.com/app-a and sampleapp.mydomain.com/app-b

I have tried the first bullet point and I can't get it to work. I can only access the first site I deploy via IP address. Even though I set the ingress up with a host name, I don't think the Gateway is getting correctly configured by the AGIC. But I am struggling on where to troubleshoot and what to do differently.

My App's Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-back
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-back
  template:
    metadata:
      labels:
        app: azure-vote-back
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: azure-vote-back
        image: redis
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 6379
          name: redis
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-back
spec:
  ports:
  - port: 6379
  selector:
    app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 2
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: azure-vote-front
        image: microsoft/azure-vote-front:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: REDIS
          value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  ports:
  - port: 80
  selector:
    app: azure-vote-front
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: voteapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - host: vote.dev.speakwrite.app
  - http:
      paths:
      - backend:
          serviceName: azure-vote-front
          servicePort: 80

My Ingress:

kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC$ kubectl get ingress
NAME           HOSTS                             ADDRESS         PORTS   AGE
newguestbook   newguestbook.dev.speakwrite.app   52.227.221.20   80      86m
voteapp        vote.dev.speakwrite.app           52.227.221.20   80      123m

Logs from AGIC Pod

kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC$ kubectl logs ingress-azure-7b996fc9fb-pkf6r
ERROR: logging before flag.Parse: I0306 17:34:01.981984       1 main.go:306] Using verbosity level 5 from environment variable APPGW_VERBOSITY_LEVEL
I0306 17:34:02.014939       1 environment.go:168] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0306 17:34:02.015044       1 main.go:149] App Gateway Details: Subscription: fad8470a-e1a9-4209-b526-011761c080a0, Resource Group: NXT-DEV-VA, Name: SW-AG01-DEV
I0306 17:34:02.015050       1 auth.go:90] Creating authorizer from Azure Managed Service Identity
I0306 17:34:02.015173       1 httpserver.go:57] Starting API Server on :8123
I0306 17:34:02.143064       1 main.go:191] Ingress Controller will observe all namespaces.
I0306 17:34:02.174564       1 context.go:129] k8s context run started
I0306 17:34:02.174620       1 context.go:168] Waiting for initial cache sync
I0306 17:34:02.274871       1 context.go:176] Initial cache sync done
I0306 17:34:02.274940       1 context.go:177] k8s context run finished
I0306 17:34:02.275033       1 worker.go:35] Worker started
I0306 17:34:02.410535       1 mutate_aks.go:78] [mutate_aks] Found IPs: map[/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP:52.227.221.20]
I0306 17:34:02.410568       1 mutate_aks.go:53] [mutate_aks] Resolving IP for ID (/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP)
I0306 17:34:02.434364       1 mutate_aks.go:60] [mutate_aks] Updated Ingress default/newguestbook IP to 52.227.221.20
I0306 17:34:02.434389       1 mutate_aks.go:53] [mutate_aks] Resolving IP for ID (/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP)
I0306 17:34:02.462937       1 context.go:438] IP 52.227.221.20 already set on Ingress default/voteapp
I0306 17:34:02.462952       1 mutate_aks.go:60] [mutate_aks] Updated Ingress default/voteapp IP to 52.227.221.20
I0306 17:34:02.493050       1 mutate_app_gateway.go:67] Existing App Gateway config: {
-- Existing App Gwy Config --    "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV",
-- Existing App Gwy Config --    "location": "usgovvirginia",
-- Existing App Gwy Config --    "properties": {
-- Existing App Gwy Config --        "backendAddressPools": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/defaultaddresspool",
-- Existing App Gwy Config --                "name": "defaultaddresspool",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "backendAddresses": [],
-- Existing App Gwy Config --                    "provisioningState": "Succeeded"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80",
-- Existing App Gwy Config --                "name": "pool-default-azure-vote-front-80-bp-80",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "backendAddresses": [
-- Existing App Gwy Config --                        {
-- Existing App Gwy Config --                            "ipAddress": "10.0.0.34"
-- Existing App Gwy Config --                        },
-- Existing App Gwy Config --                        {
-- Existing App Gwy Config --                            "ipAddress": "10.0.0.9"
-- Existing App Gwy Config --                        }
-- Existing App Gwy Config --                    ],
-- Existing App Gwy Config --                    "provisioningState": "Succeeded"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "backendHttpSettingsCollection": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp",
-- Existing App Gwy Config --                "name": "bp-default-azure-vote-front-80-80-voteapp",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "cookieBasedAffinity": "Disabled",
-- Existing App Gwy Config --                    "pickHostNameFromBackendAddress": false,
-- Existing App Gwy Config --                    "port": 80,
-- Existing App Gwy Config --                    "probe": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "requestTimeout": 30
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/defaulthttpsetting",
-- Existing App Gwy Config --                "name": "defaulthttpsetting",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "cookieBasedAffinity": "Disabled",
-- Existing App Gwy Config --                    "pickHostNameFromBackendAddress": false,
-- Existing App Gwy Config --                    "port": 80,
-- Existing App Gwy Config --                    "probe": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "requestTimeout": 30
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "frontendIPConfigurations": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP",
-- Existing App Gwy Config --                "name": "appGatewayFrontendIP",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "privateIPAllocationMethod": "Dynamic",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "publicIPAddress": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/publicIPAddresses/SW-GW-PIP01-DEV"
-- Existing App Gwy Config --                    }
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "frontendPorts": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort", 
-- Existing App Gwy Config --                "name": "httpPort",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "port": 80,
-- Existing App Gwy Config --                    "provisioningState": "Succeeded"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "gatewayIPConfigurations": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/gatewayIPConfigurations/appGatewayIpConfig",
-- Existing App Gwy Config --                "name": "appGatewayIpConfig",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "subnet": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/virtualNetworks/SW-VNET01-DEV/subnets/appgwsubnet"
-- Existing App Gwy Config --                    }
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "httpListeners": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "name": "fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "frontendIPConfiguration": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "frontendPort": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "hostnames": [],
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "requireServerNameIndication": false
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "operationalState": "Running",
-- Existing App Gwy Config --        "probes": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http",
-- Existing App Gwy Config --                "name": "defaultprobe-Http",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "host": "localhost",
-- Existing App Gwy Config --                    "interval": 30,
-- Existing App Gwy Config --                    "match": {},
-- Existing App Gwy Config --                    "minServers": 0,
-- Existing App Gwy Config --                    "path": "/",
-- Existing App Gwy Config --                    "pickHostNameFromBackendHttpSettings": false,
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "timeout": 30,
-- Existing App Gwy Config --                    "unhealthyThreshold": 3
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Https",
-- Existing App Gwy Config --                "name": "defaultprobe-Https",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "host": "localhost",
-- Existing App Gwy Config --                    "interval": 30,
-- Existing App Gwy Config --                    "match": {},
-- Existing App Gwy Config --                    "minServers": 0,
-- Existing App Gwy Config --                    "path": "/",
-- Existing App Gwy Config --                    "pickHostNameFromBackendHttpSettings": false,
-- Existing App Gwy Config --                    "protocol": "Https",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "timeout": 30,
-- Existing App Gwy Config --                    "unhealthyThreshold": 3
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp",
-- Existing App Gwy Config --                "name": "pb-default-azure-vote-front-80-voteapp",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "host": "localhost",
-- Existing App Gwy Config --                    "interval": 30,
-- Existing App Gwy Config --                    "match": {},
-- Existing App Gwy Config --                    "minServers": 0,
-- Existing App Gwy Config --                    "path": "/",
-- Existing App Gwy Config --                    "pickHostNameFromBackendHttpSettings": false,
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "timeout": 30,
-- Existing App Gwy Config --                    "unhealthyThreshold": 3
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "provisioningState": "Succeeded",
-- Existing App Gwy Config --        "redirectConfigurations": [],
-- Existing App Gwy Config --        "requestRoutingRules": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/requestRoutingRules/rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "name": "rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "backendAddressPool": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "backendHttpSettings": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "httpListener": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "ruleType": "Basic"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "resourceGuid": "5ed921cf-f072-4e7e-99a6-bb602e02e1ea",
-- Existing App Gwy Config --        "rewriteRuleSets": [],
-- Existing App Gwy Config --        "sku": {
-- Existing App Gwy Config --            "capacity": 2,
-- Existing App Gwy Config --            "name": "WAF_v2",
-- Existing App Gwy Config --            "tier": "WAF_v2"
-- Existing App Gwy Config --        },
-- Existing App Gwy Config --        "trustedRootCertificates": [],
-- Existing App Gwy Config --        "urlPathMaps": [],
-- Existing App Gwy Config --        "webApplicationFirewallConfiguration": {
-- Existing App Gwy Config --            "disabledRuleGroups": [],
-- Existing App Gwy Config --            "enabled": true,
-- Existing App Gwy Config --            "fileUploadLimitInMb": 100,
-- Existing App Gwy Config --            "firewallMode": "Detection",
-- Existing App Gwy Config --            "maxRequestBodySizeInKb": 128,
-- Existing App Gwy Config --            "requestBodyCheck": true,
-- Existing App Gwy Config --            "ruleSetType": "OWASP",
-- Existing App Gwy Config --            "ruleSetVersion": "3.0"
-- Existing App Gwy Config --        }
-- Existing App Gwy Config --    },
-- Existing App Gwy Config --    "tags": {
-- Existing App Gwy Config --        "ingress-for-aks-cluster-id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourcegroups/nxt-dev-va/providers/Microsoft.ContainerService/managedClusters/sw-aks01-dev",  
-- Existing App Gwy Config --        "last-updated-by-k8s-ingress": "2020-03-06 16:57:41.950229032 +0000 UTC m=+261.042702839",
-- Existing App Gwy Config --        "managed-by-k8s-ingress": "1.0.1-rc2/2105e9e7/2019-12-10-07:01T+0000"
-- Existing App Gwy Config --    }
-- Existing App Gwy Config --}
I0306 17:34:02.493102       1 health_probes.go:55] Created default HTTP probe defaultprobe-Http
I0306 17:34:02.493108       1 health_probes.go:56] Created default HTTPS probe defaultprobe-Http
I0306 17:34:02.493114       1 ingress_rules.go:106] [default] Skip rule #1 for host 'newguestbook.dev.speakwrite.app' - it has no HTTP rules.
I0306 17:34:02.493120       1 ingress_rules.go:112] Found backend:default/frontend
I0306 17:34:02.493126       1 ingress_rules.go:106] [default] Skip rule #1 for host 'vote.dev.speakwrite.app' - it has no HTTP rules.
I0306 17:34:02.493131       1 ingress_rules.go:112] Found backend:default/azure-vote-front
I0306 17:34:02.493214       1 health_probes.go:70] Created probe pb-default-frontend-80-newguestbook for ingress default/newguestbook and service default/frontend
I0306 17:34:02.493270       1 health_probes.go:70] Created probe pb-default-azure-vote-front-80-voteapp for ingress default/voteapp and service default/azure-vote-front
I0306 17:34:02.493297       1 backendhttpsettings.go:185] Created backend http settings bp-default-azure-vote-front-80-80-voteapp for ingress default/voteapp and service default/azure-vote-front
I0306 17:34:02.493309       1 backendhttpsettings.go:185] Created backend http settings bp-default-frontend-80-80-newguestbook for ingress default/newguestbook and service default/frontend
I0306 17:34:02.493318       1 backendaddresspools.go:37] Created default backend pool defaultaddresspool
I0306 17:34:02.493331       1 backendaddresspools.go:48] Created backend pool pool-default-azure-vote-front-80-bp-80 for service default/azure-vote-front
I0306 17:34:02.493339       1 frontend_listeners.go:117] Processing Rules for Ingress: default/newguestbook
I0306 17:34:02.493348       1 frontend_listeners.go:117] Processing Rules for Ingress: default/voteapp
I0306 17:34:02.493464       1 requestroutingrules.go:112] Bound basic rule: rr-6d1d6d2bd4405b8228172c2ef8a065fb to listener: fl-6d1d6d2bd4405b8228172c2ef8a065fb ([    ], 80) for backend pool pool-default-azure-vote-front-80-bp-80 and backend http settings bp-default-azure-vote-front-80-80-voteapp
I0306 17:34:02.499928       1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0306 17:34:22.988143       1 mutate_app_gateway.go:178] {
-- App Gwy config --    "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV",
-- App Gwy config --    "location": "usgovvirginia",
-- App Gwy config --    "properties": {
-- App Gwy config --        "backendAddressPools": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/defaultaddresspool",
-- App Gwy config --                "name": "defaultaddresspool",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddresses": []
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80",
-- App Gwy config --                "name": "pool-default-azure-vote-front-80-bp-80",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddresses": [
-- App Gwy config --                        {
-- App Gwy config --                            "ipAddress": "10.0.0.34"
-- App Gwy config --                        },
-- App Gwy config --                        {
-- App Gwy config --                            "ipAddress": "10.0.0.9"
-- App Gwy config --                        }
-- App Gwy config --                    ]
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "backendHttpSettingsCollection": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp",
-- App Gwy config --                "name": "bp-default-azure-vote-front-80-80-voteapp",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "probe": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp"
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-frontend-80-80-newguestbook",
-- App Gwy config --                "name": "bp-default-frontend-80-80-newguestbook",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "probe": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-frontend-80-newguestbook"
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/defaulthttpsetting",
-- App Gwy config --                "name": "defaulthttpsetting",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "probe": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http" 
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "frontendIPConfigurations": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP",
-- App Gwy config --                "name": "appGatewayFrontendIP",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "privateIPAllocationMethod": "Dynamic",
-- App Gwy config --                    "provisioningState": "Succeeded",
-- App Gwy config --                    "publicIPAddress": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/publicIPAddresses/SW-GW-PIP01-DEV"
-- App Gwy config --                    }
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "frontendPorts": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort",
-- App Gwy config --                "name": "httpPort",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "provisioningState": "Succeeded"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "gatewayIPConfigurations": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/gatewayIPConfigurations/appGatewayIpConfig",
-- App Gwy config --                "name": "appGatewayIpConfig",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "provisioningState": "Succeeded",
-- App Gwy config --                    "subnet": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/virtualNetworks/SW-VNET01-DEV/subnets/appgwsubnet"        
-- App Gwy config --                    }
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "httpListeners": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "name": "fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "frontendIPConfiguration": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP"
-- App Gwy config --                    },
-- App Gwy config --                    "frontendPort": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort"   
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "operationalState": "Running",
-- App Gwy config --        "probes": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http",        
-- App Gwy config --                "name": "defaultprobe-Http",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Http",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Https",       
-- App Gwy config --                "name": "defaultprobe-Https",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Https",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp",
-- App Gwy config --                "name": "pb-default-azure-vote-front-80-voteapp",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Http",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-frontend-80-newguestbook",
-- App Gwy config --                "name": "pb-default-frontend-80-newguestbook",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Http",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "provisioningState": "Succeeded",
-- App Gwy config --        "redirectConfigurations": null,
-- App Gwy config --        "requestRoutingRules": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/requestRoutingRules/rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "name": "rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddressPool": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80"
-- App Gwy config --                    },
-- App Gwy config --                    "backendHttpSettings": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp"
-- App Gwy config --                    },
-- App Gwy config --                    "httpListener": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb"
-- App Gwy config --                    },
-- App Gwy config --                    "ruleType": "Basic"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "resourceGuid": "5ed921cf-f072-4e7e-99a6-bb602e02e1ea",
-- App Gwy config --        "rewriteRuleSets": [],
-- App Gwy config --        "sku": {
-- App Gwy config --            "capacity": 2,
-- App Gwy config --            "name": "WAF_v2",
-- App Gwy config --            "tier": "WAF_v2"
-- App Gwy config --        },
-- App Gwy config --        "trustedRootCertificates": [],
-- App Gwy config --        "urlPathMaps": null,
-- App Gwy config --        "webApplicationFirewallConfiguration": {
-- App Gwy config --            "disabledRuleGroups": [],
-- App Gwy config --            "enabled": true,
-- App Gwy config --            "fileUploadLimitInMb": 100,
-- App Gwy config --            "firewallMode": "Detection",
-- App Gwy config --            "maxRequestBodySizeInKb": 128,
-- App Gwy config --            "requestBodyCheck": true,
-- App Gwy config --            "ruleSetType": "OWASP",
-- App Gwy config --            "ruleSetVersion": "3.0"
-- App Gwy config --        }
-- App Gwy config --    },
-- App Gwy config --    "tags": {
-- App Gwy config --        "ingress-for-aks-cluster-id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourcegroups/nxt-dev-va/providers/Microsoft.ContainerService/managedClusters/sw-aks01-dev",
-- App Gwy config --        "last-updated-by-k8s-ingress": "2020-03-06 17:34:02.499410253 +0000 UTC m=+0.523574564",
-- App Gwy config --        "managed-by-k8s-ingress": "1.0.1-rc2/2105e9e7/2019-12-10-07:01T+0000"
-- App Gwy config --    }
-- App Gwy config --}
I0306 17:34:22.988187       1 mutate_app_gateway.go:182] Applied App Gateway config in 20.488238365s
I0306 17:34:22.988199       1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0306 17:34:22.988713       1 mutate_app_gateway.go:203] END AppGateway deployment
I0306 17:34:24.048705       1 mutate_aks.go:78] [mutate_aks] Found IPs: map[/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP:52.227.221.20]
I0306 17:34:24.048735       1 mutate_aks.go:53] [mutate_aks] Resolving IP for ID (/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP)
I0306 17:34:24.052265       1 context.go:438] IP 52.227.221.20 already set on Ingress default/newguestbook
I0306 17:34:24.052332       1 mutate_aks.go:60] [mutate_aks] Updated Ingress default/newguestbook IP to 52.227.221.20
I0306 17:34:24.052351       1 mutate_aks.go:53] [mutate_aks] Resolving IP for ID (/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP)
I0306 17:34:24.055273       1 context.go:438] IP 52.227.221.20 already set on Ingress default/voteapp
I0306 17:34:24.055301       1 mutate_aks.go:60] [mutate_aks] Updated Ingress default/voteapp IP to 52.227.221.20
I0306 17:34:24.087470       1 mutate_app_gateway.go:67] Existing App Gateway config: {
-- Existing App Gwy Config --    "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV",
-- Existing App Gwy Config --    "location": "usgovvirginia",
-- Existing App Gwy Config --    "properties": {
-- Existing App Gwy Config --        "backendAddressPools": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/defaultaddresspool",
-- Existing App Gwy Config --                "name": "defaultaddresspool",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "backendAddresses": [],
-- Existing App Gwy Config --                    "provisioningState": "Succeeded"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80",
-- Existing App Gwy Config --                "name": "pool-default-azure-vote-front-80-bp-80",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "backendAddresses": [
-- Existing App Gwy Config --                        {
-- Existing App Gwy Config --                            "ipAddress": "10.0.0.34"
-- Existing App Gwy Config --                        },
-- Existing App Gwy Config --                        {
-- Existing App Gwy Config --                            "ipAddress": "10.0.0.9"
-- Existing App Gwy Config --                        }
-- Existing App Gwy Config --                    ],
-- Existing App Gwy Config --                    "provisioningState": "Succeeded"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "backendHttpSettingsCollection": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp",
-- Existing App Gwy Config --                "name": "bp-default-azure-vote-front-80-80-voteapp",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "cookieBasedAffinity": "Disabled",
-- Existing App Gwy Config --                    "pickHostNameFromBackendAddress": false,
-- Existing App Gwy Config --                    "port": 80,
-- Existing App Gwy Config --                    "probe": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "requestTimeout": 30
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-frontend-80-80-newguestbook",
-- Existing App Gwy Config --                "name": "bp-default-frontend-80-80-newguestbook",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "cookieBasedAffinity": "Disabled",
-- Existing App Gwy Config --                    "pickHostNameFromBackendAddress": false,
-- Existing App Gwy Config --                    "port": 80,
-- Existing App Gwy Config --                    "probe": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-frontend-80-newguestbook"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "requestTimeout": 30
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/defaulthttpsetting",
-- Existing App Gwy Config --                "name": "defaulthttpsetting",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "cookieBasedAffinity": "Disabled",
-- Existing App Gwy Config --                    "pickHostNameFromBackendAddress": false,
-- Existing App Gwy Config --                    "port": 80,
-- Existing App Gwy Config --                    "probe": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "requestTimeout": 30
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "frontendIPConfigurations": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP",
-- Existing App Gwy Config --                "name": "appGatewayFrontendIP",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "privateIPAllocationMethod": "Dynamic",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "publicIPAddress": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/publicIPAddresses/SW-GW-PIP01-DEV"
-- Existing App Gwy Config --                    }
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "frontendPorts": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort", 
-- Existing App Gwy Config --                "name": "httpPort",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "port": 80,
-- Existing App Gwy Config --                    "provisioningState": "Succeeded"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "gatewayIPConfigurations": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/gatewayIPConfigurations/appGatewayIpConfig",
-- Existing App Gwy Config --                "name": "appGatewayIpConfig",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "subnet": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/virtualNetworks/SW-VNET01-DEV/subnets/appgwsubnet"
-- Existing App Gwy Config --                    }
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "httpListeners": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "name": "fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "frontendIPConfiguration": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "frontendPort": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "hostnames": [],
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "requireServerNameIndication": false
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "operationalState": "Running",
-- Existing App Gwy Config --        "probes": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http",
-- Existing App Gwy Config --                "name": "defaultprobe-Http",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "host": "localhost",
-- Existing App Gwy Config --                    "interval": 30,
-- Existing App Gwy Config --                    "match": {},
-- Existing App Gwy Config --                    "minServers": 0,
-- Existing App Gwy Config --                    "path": "/",
-- Existing App Gwy Config --                    "pickHostNameFromBackendHttpSettings": false,
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "timeout": 30,
-- Existing App Gwy Config --                    "unhealthyThreshold": 3
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Https",
-- Existing App Gwy Config --                "name": "defaultprobe-Https",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "host": "localhost",
-- Existing App Gwy Config --                    "interval": 30,
-- Existing App Gwy Config --                    "match": {},
-- Existing App Gwy Config --                    "minServers": 0,
-- Existing App Gwy Config --                    "path": "/",
-- Existing App Gwy Config --                    "pickHostNameFromBackendHttpSettings": false,
-- Existing App Gwy Config --                    "protocol": "Https",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "timeout": 30,
-- Existing App Gwy Config --                    "unhealthyThreshold": 3
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp",
-- Existing App Gwy Config --                "name": "pb-default-azure-vote-front-80-voteapp",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "host": "localhost",
-- Existing App Gwy Config --                    "interval": 30,
-- Existing App Gwy Config --                    "match": {},
-- Existing App Gwy Config --                    "minServers": 0,
-- Existing App Gwy Config --                    "path": "/",
-- Existing App Gwy Config --                    "pickHostNameFromBackendHttpSettings": false,
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "timeout": 30,
-- Existing App Gwy Config --                    "unhealthyThreshold": 3
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            },
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-frontend-80-newguestbook",
-- Existing App Gwy Config --                "name": "pb-default-frontend-80-newguestbook",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "host": "localhost",
-- Existing App Gwy Config --                    "interval": 30,
-- Existing App Gwy Config --                    "match": {},
-- Existing App Gwy Config --                    "minServers": 0,
-- Existing App Gwy Config --                    "path": "/",
-- Existing App Gwy Config --                    "pickHostNameFromBackendHttpSettings": false,
-- Existing App Gwy Config --                    "protocol": "Http",
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "timeout": 30,
-- Existing App Gwy Config --                    "unhealthyThreshold": 3
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "provisioningState": "Succeeded",
-- Existing App Gwy Config --        "redirectConfigurations": [],
-- Existing App Gwy Config --        "requestRoutingRules": [
-- Existing App Gwy Config --            {
-- Existing App Gwy Config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/requestRoutingRules/rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "name": "rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- Existing App Gwy Config --                "properties": {
-- Existing App Gwy Config --                    "backendAddressPool": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "backendHttpSettings": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "httpListener": {
-- Existing App Gwy Config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb"
-- Existing App Gwy Config --                    },
-- Existing App Gwy Config --                    "provisioningState": "Succeeded",
-- Existing App Gwy Config --                    "ruleType": "Basic"
-- Existing App Gwy Config --                }
-- Existing App Gwy Config --            }
-- Existing App Gwy Config --        ],
-- Existing App Gwy Config --        "resourceGuid": "5ed921cf-f072-4e7e-99a6-bb602e02e1ea",
-- Existing App Gwy Config --        "rewriteRuleSets": [],
-- Existing App Gwy Config --        "sku": {
-- Existing App Gwy Config --            "capacity": 2,
-- Existing App Gwy Config --            "name": "WAF_v2",
-- Existing App Gwy Config --            "tier": "WAF_v2"
-- Existing App Gwy Config --        },
-- Existing App Gwy Config --        "trustedRootCertificates": [],
-- Existing App Gwy Config --        "urlPathMaps": [],
-- Existing App Gwy Config --        "webApplicationFirewallConfiguration": {
-- Existing App Gwy Config --            "disabledRuleGroups": [],
-- Existing App Gwy Config --            "enabled": true,
-- Existing App Gwy Config --            "fileUploadLimitInMb": 100,
-- Existing App Gwy Config --            "firewallMode": "Detection",
-- Existing App Gwy Config --            "maxRequestBodySizeInKb": 128,
-- Existing App Gwy Config --            "requestBodyCheck": true,
-- Existing App Gwy Config --            "ruleSetType": "OWASP",
-- Existing App Gwy Config --            "ruleSetVersion": "3.0"
-- Existing App Gwy Config --        }
-- Existing App Gwy Config --    },
-- Existing App Gwy Config --    "tags": {
-- Existing App Gwy Config --        "ingress-for-aks-cluster-id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourcegroups/nxt-dev-va/providers/Microsoft.ContainerService/managedClusters/sw-aks01-dev",  
-- Existing App Gwy Config --        "last-updated-by-k8s-ingress": "2020-03-06 17:34:02.499410253 +0000 UTC m=+0.523574564",
-- Existing App Gwy Config --        "managed-by-k8s-ingress": "1.0.1-rc2/2105e9e7/2019-12-10-07:01T+0000"
-- Existing App Gwy Config --    }
-- Existing App Gwy Config --}
I0306 17:34:24.087536       1 health_probes.go:55] Created default HTTP probe defaultprobe-Http
I0306 17:34:24.087544       1 health_probes.go:56] Created default HTTPS probe defaultprobe-Http
I0306 17:34:24.087551       1 ingress_rules.go:106] [default] Skip rule #1 for host 'newguestbook.dev.speakwrite.app' - it has no HTTP rules.
I0306 17:34:24.087559       1 ingress_rules.go:112] Found backend:default/frontend
I0306 17:34:24.087565       1 ingress_rules.go:106] [default] Skip rule #1 for host 'vote.dev.speakwrite.app' - it has no HTTP rules.
I0306 17:34:24.087570       1 ingress_rules.go:112] Found backend:default/azure-vote-front
I0306 17:34:24.087658       1 health_probes.go:70] Created probe pb-default-frontend-80-newguestbook for ingress default/newguestbook and service default/frontend
I0306 17:34:24.087720       1 health_probes.go:70] Created probe pb-default-azure-vote-front-80-voteapp for ingress default/voteapp and service default/azure-vote-front
I0306 17:34:24.087752       1 backendhttpsettings.go:185] Created backend http settings bp-default-frontend-80-80-newguestbook for ingress default/newguestbook and service default/frontend
I0306 17:34:24.087769       1 backendhttpsettings.go:185] Created backend http settings bp-default-azure-vote-front-80-80-voteapp for ingress default/voteapp and service default/azure-vote-front
I0306 17:34:24.087778       1 backendaddresspools.go:37] Created default backend pool defaultaddresspool
I0306 17:34:24.087793       1 backendaddresspools.go:48] Created backend pool pool-default-frontend-80-bp-80 for service default/frontend
I0306 17:34:24.087808       1 backendaddresspools.go:48] Created backend pool pool-default-azure-vote-front-80-bp-80 for service default/azure-vote-front
I0306 17:34:24.087816       1 frontend_listeners.go:117] Processing Rules for Ingress: default/newguestbook
I0306 17:34:24.087825       1 frontend_listeners.go:117] Processing Rules for Ingress: default/voteapp
I0306 17:34:24.088022       1 requestroutingrules.go:112] Bound basic rule: rr-6d1d6d2bd4405b8228172c2ef8a065fb to listener: fl-6d1d6d2bd4405b8228172c2ef8a065fb ([    ], 80) for backend pool pool-default-azure-vote-front-80-bp-80 and backend http settings bp-default-azure-vote-front-80-80-voteapp
I0306 17:34:24.093584       1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0306 17:34:44.446462       1 mutate_app_gateway.go:178] {
-- App Gwy config --    "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV",
-- App Gwy config --    "location": "usgovvirginia",
-- App Gwy config --    "properties": {
-- App Gwy config --        "backendAddressPools": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/defaultaddresspool",
-- App Gwy config --                "name": "defaultaddresspool",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddresses": []
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80",
-- App Gwy config --                "name": "pool-default-azure-vote-front-80-bp-80",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddresses": [
-- App Gwy config --                        {
-- App Gwy config --                            "ipAddress": "10.0.0.34"
-- App Gwy config --                        },
-- App Gwy config --                        {
-- App Gwy config --                            "ipAddress": "10.0.0.9"
-- App Gwy config --                        }
-- App Gwy config --                    ]
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-frontend-80-bp-80",
-- App Gwy config --                "name": "pool-default-frontend-80-bp-80",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddresses": [
-- App Gwy config --                        {
-- App Gwy config --                            "ipAddress": "10.0.0.18"
-- App Gwy config --                        },
-- App Gwy config --                        {
-- App Gwy config --                            "ipAddress": "10.0.0.7"
-- App Gwy config --                        },
-- App Gwy config --                        {
-- App Gwy config --                            "ipAddress": "10.0.0.8"
-- App Gwy config --                        }
-- App Gwy config --                    ]
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "backendHttpSettingsCollection": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp",
-- App Gwy config --                "name": "bp-default-azure-vote-front-80-80-voteapp",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "probe": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp"
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-frontend-80-80-newguestbook",
-- App Gwy config --                "name": "bp-default-frontend-80-80-newguestbook",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "probe": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-frontend-80-newguestbook"
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/defaulthttpsetting",
-- App Gwy config --                "name": "defaulthttpsetting",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "probe": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http"
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "frontendIPConfigurations": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP",
-- App Gwy config --                "name": "appGatewayFrontendIP",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "privateIPAllocationMethod": "Dynamic",
-- App Gwy config --                    "provisioningState": "Succeeded",
-- App Gwy config --                    "publicIPAddress": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/publicIPAddresses/SW-GW-PIP01-DEV"
-- App Gwy config --                    }
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "frontendPorts": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort",
-- App Gwy config --                "name": "httpPort",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "port": 80,
-- App Gwy config --                    "provisioningState": "Succeeded"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "gatewayIPConfigurations": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/gatewayIPConfigurations/appGatewayIpConfig",
-- App Gwy config --                "name": "appGatewayIpConfig",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "provisioningState": "Succeeded",
-- App Gwy config --                    "subnet": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/virtualNetworks/SW-VNET01-DEV/subnets/appgwsubnet"        
-- App Gwy config --                    }
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "httpListeners": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "name": "fl-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "frontendIPConfiguration": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendIPConfigurations/appGatewayFrontendIP"
-- App Gwy config --                    },
-- App Gwy config --                    "frontendPort": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/frontendPorts/httpPort"   
-- App Gwy config --                    },
-- App Gwy config --                    "protocol": "Http"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "operationalState": "Running",
-- App Gwy config --        "probes": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Http",        
-- App Gwy config --                "name": "defaultprobe-Http",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Http",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/defaultprobe-Https",       
-- App Gwy config --                "name": "defaultprobe-Https",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Https",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-azure-vote-front-80-voteapp",
-- App Gwy config --                "name": "pb-default-azure-vote-front-80-voteapp",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Http",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            },
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/probes/pb-default-frontend-80-newguestbook",
-- App Gwy config --                "name": "pb-default-frontend-80-newguestbook",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "host": "localhost",
-- App Gwy config --                    "interval": 30,
-- App Gwy config --                    "path": "/",
-- App Gwy config --                    "protocol": "Http",
-- App Gwy config --                    "timeout": 30,
-- App Gwy config --                    "unhealthyThreshold": 3
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "provisioningState": "Succeeded",
-- App Gwy config --        "redirectConfigurations": null,
-- App Gwy config --        "requestRoutingRules": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/requestRoutingRules/rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "name": "rr-6d1d6d2bd4405b8228172c2ef8a065fb",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddressPool": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendAddressPools/pool-default-azure-vote-front-80-bp-80"
-- App Gwy config --                    },
-- App Gwy config --                    "backendHttpSettings": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/backendHttpSettingsCollection/bp-default-azure-vote-front-80-80-voteapp"
-- App Gwy config --                    },
-- App Gwy config --                    "httpListener": {
-- App Gwy config --                        "id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourceGroups/NXT-DEV-VA/providers/Microsoft.Network/applicationGateways/SW-AG01-DEV/httpListeners/fl-6d1d6d2bd4405b8228172c2ef8a065fb"
-- App Gwy config --                    },
-- App Gwy config --                    "ruleType": "Basic"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],
-- App Gwy config --        "resourceGuid": "5ed921cf-f072-4e7e-99a6-bb602e02e1ea",
-- App Gwy config --        "rewriteRuleSets": [],
-- App Gwy config --        "sku": {
-- App Gwy config --            "capacity": 2,
-- App Gwy config --            "name": "WAF_v2",
-- App Gwy config --            "tier": "WAF_v2"
-- App Gwy config --        },
-- App Gwy config --        "trustedRootCertificates": [],
-- App Gwy config --        "urlPathMaps": null,
-- App Gwy config --        "webApplicationFirewallConfiguration": {
-- App Gwy config --            "disabledRuleGroups": [],
-- App Gwy config --            "enabled": true,
-- App Gwy config --            "fileUploadLimitInMb": 100,
-- App Gwy config --            "firewallMode": "Detection",
-- App Gwy config --            "maxRequestBodySizeInKb": 128,
-- App Gwy config --            "requestBodyCheck": true,
-- App Gwy config --            "ruleSetType": "OWASP",
-- App Gwy config --            "ruleSetVersion": "3.0"
-- App Gwy config --        }
-- App Gwy config --    },
-- App Gwy config --    "tags": {
-- App Gwy config --        "ingress-for-aks-cluster-id": "/subscriptions/fad8470a-e1a9-4209-b526-011761c080a0/resourcegroups/nxt-dev-va/providers/Microsoft.ContainerService/managedClusters/sw-aks01-dev",
-- App Gwy config --        "last-updated-by-k8s-ingress": "2020-03-06 17:34:24.093030767 +0000 UTC m=+22.117195078",
-- App Gwy config --        "managed-by-k8s-ingress": "1.0.1-rc2/2105e9e7/2019-12-10-07:01T+0000"
-- App Gwy config --    }
-- App Gwy config --}
I0306 17:34:44.446498       1 mutate_app_gateway.go:182] Applied App Gateway config in 20.352891622s
I0306 17:34:44.446509       1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0306 17:34:44.447097       1 mutate_app_gateway.go:203] END AppGateway deployment

akshaysngupta commented 4 years ago

@flyer299 You need to make a minor change to your ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: voteapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - host: vote.dev.speakwrite.app
  - http: # remove the the dash before http
      paths:
      - backend:
          serviceName: azure-vote-front
          servicePort: 80

Please remove the dash before http. A dash means a new item in the list. So the host and http in your are considered two separate rules instead of one. You can use this tutorial for reference.

akshaysngupta commented 4 years ago

Thanks for pointing. Here is an example how you can do path-based routing.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: voteapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - host: vote.dev.speakwrite.app
    http:
      paths:
      - path: /app-a
        backend:
          serviceName: app-a
          servicePort: 80
      - path: /app-b
        backend:
          serviceName: app-b
          servicePort: 80

flyer299 commented 4 years ago

That helped me move forward a bit... But now I am having a problem once I've added LetsEncrypt to the mix.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: guestbook-letsencrypt-staging
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-prod
    cert-manager.io/acme-challenge-type: http01
spec:
  tls:
  - hosts:
    - guestbook.dev.speakwrite.app
    secretName: guestbook-secret-name
  rules:
  - host: guestbook.dev.speakwrite.app
    http:
      paths:
      - backend:
          serviceName: frontend
          servicePort: 80

The AGIC Pod now gives this error:

Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"guestbook-letsencrypt-staging", UID:"7f53c972-cc12-4b42-a662-5293c3c3ad00", APIVersion:"extensions/v1beta1", ResourceVersion:"3024037", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [default/guestbook-secret-name]

I saw this issue: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/721 which is the same error I am getting. But the resolution to that issue has already been applied in my examples.

What is causing the SSL to not work?

Here is my ClusterIssuer

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: itaccounts@speakwrite.com
    # ACME server URL for Let’s Encrypt’s staging environment.
    # The staging environment will not issue trusted certificates but is
    # used to ensure that the verification process is working properly
    # before moving to production
    server: https://acme-v01.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-secret
    # Enable the HTTP-01 challenge provider
    # you prove ownership of a domain by ensuring that a particular
    # file is present at the domain
    solvers:
    - http01:
        ingress:
            class: azure/application-gateway

I have these items running in the cert-manager namespace:

kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC/LetsEncrypt$ kubectl get services --namespace cert-manager
NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
cert-manager           ClusterIP   10.2.33.11    <none>        9402/TCP   85m
cert-manager-webhook   ClusterIP   10.2.97.176   <none>        443/TCP    85m
kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC/LetsEncrypt$ kubectl get deployments --namespace cert-manager
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           86m
cert-manager-cainjector   1/1     1            1           86m
cert-manager-webhook      1/1     1            1           86m
kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC/LetsEncrypt$

akshaysngupta commented 4 years ago

@flyer299 Were you able to get past this issue ? Did you check if the secret exists ?

flyer299 commented 4 years ago

I was never able to get Let’s Encrypt working with Application Gateway. I had to revert back to an NGIX Ingress Controller. I am going to wait a month or two and check back on Application Gateway. I really want App Gateway for our solution when we go to production, but I want to give the teams behind it more time to make the integration with Government cloud more robust and straight forward before I invest any more of our time on it.

From: Akshay Gupta notifications@github.com Sent: Friday, April 3, 2020 7:56 PM To: Azure/application-gateway-kubernetes-ingress application-gateway-kubernetes-ingress@noreply.github.com Cc: Ken VeArd ken@veard.net; Mention mention@noreply.github.com Subject: Re: [Azure/application-gateway-kubernetes-ingress] Comprehensive Reference/Documentation (#758)

@flyer299https://github.com/flyer299 Were you able to get past this issue ? Did you check if the secret exists ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Azure/application-gateway-kubernetes-ingress/issues/758#issuecomment-608934691, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABURFKGJ7YMYTPYOMUZYEE3RK2AR5ANCNFSM4LDF7XFA.

thatguycalledrob commented 3 years ago

I am facing the same issue.

I am trying to install LetsEncrypt via helm into a terraform deployed cluster using gateway. Here are the relevant bits of the terraform code:

resource "kubernetes_namespace" "agic_ns" {
  metadata {
    name = "agic"
  }
}

resource "helm_release" "agic" {
  name       = "agic"
  repository = "https://appgwingress.blob.core.windows.net/ingress-azure-helm-package"
  chart      = "ingress-azure"
  namespace  = kubernetes_namespace.agic_ns.metadata.0.name
  version    = "1.2.0"

  set {
    name  = "verbosityLevel"
    value = 5
  }
  set {
      ... lots of irrelevant stuff here ...
  }
}

resource "kubernetes_namespace" "cert_ns" {
  metadata {
    name = "cert-manager"
  }
}

resource "helm_release" "cert_manager" {
  repository = "https://charts.jetstack.io"
  name       = "cert-manager"
  chart      = "cert-manager"
  namespace  =kubernetes_namespace.cert_ns.metadata.0.name
  version    = "v1.0.2"
  set {
    name  = "installCRDs"
    value = true
  }
}

Firstly, I try to deploy the ingress with a static passwordless pks cert in a keyvault (pod identity helm used here). THIS WORKS!

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: my-svc-ns
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: my-cert
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: www.example.com
    http:
     paths:
     ... paths here ...

Next, I make some minor changes in line with the documentation found here: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/lets-encrypt.md

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: my-svc-ns
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-cert
    cert-manager.io/acme-challenge-type: http01
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - www.example.com
    secretName: letsencrypt-secret
  rules:
  - host: www.example.com
    http:
     paths:
     ... paths here ...

---

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-cert
  namespace: my-svc-ns
spec:
  acme:
    email: email@example.com

    server: https://acme-staging-v02.api.letsencrypt.org/directory # testing certs
    # server: https://acme-v02.api.letsencrypt.org/directory # prod certs

    privateKeySecretRef:
      name: letsencrypt-secret

    solvers:
      - http01:
          ingress:
            class: azure/application-gateway

Annoyingly this breaks the app gateway. If I hit http://my-site, I get a 307 and pointed at https://my-site which returns a cannot be reached (not an application-gateway 502 or 404 page!) Inspecting the app gateway layer I get the following error:

frontend_listeners.go:121] Processing Rules for Ingress: my-svc-ns/ingress
Event(v1.ObjectReference{Kind:"Ingress", Namespace:"my-svc-ns", Name:"ingress", UID:"c4cc4899-1dc4-4a23-83d1-da95e54b280c", APIVersion:"extensions/v1beta1", ResourceVersion:"3170126", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: ][my-svc-ns/letsencrypt-secret]

I can inspect the issuer via kubectl describe clusterissuer -A and it seems to have generated me a (something?) certificate: Everything looks OK here.

API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
... stuff here ...
Spec:
  Acme:
    Email:           example@example.com
    Preferred Chain:
    Private Key Secret Ref:
      Name:  letsencrypt-secret
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  azure/application-gateway
Status:
  Acme:
    Last Registered Email:  example@example.com
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/16112602
  Conditions:
    Last Transition Time:  2020-10-14T12:18:34Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready

I can inspect the secrets via: kubectl get secrets -n cert-manager and I get the following:

cert-manager      cert-manager-cainjector-token-ms887              kubernetes.io/service-account-token   3      20h
cert-manager      cert-manager-token-gr2rt                         kubernetes.io/service-account-token   3      20h
cert-manager      cert-manager-webhook-ca                          Opaque                                3      20h
cert-manager      cert-manager-webhook-token-lgd4n                 kubernetes.io/service-account-token   3      20h
cert-manager      default-token-cdtws                              kubernetes.io/service-account-token   3      20h
cert-manager      letsencrypt-secret                               Opaque                                1      19h
cert-manager      sh.helm.release.v1.cert-manager.v1               helm.sh/release.v1                    1      20h

Note that via kubectl describe secret -n cert-manager letsencrypt-secret I can see that a .key file has been created:

Name:         letsencrypt-secret
Namespace:    cert-manager
Type:  Opaque
Data
====
tls.key:  1675 bytes

Next up, as per this Microsoft documentation: https://docs.microsoft.com/en-us/azure/aks/ingress-static-ip I inspect the certificates via kubectl get certificates -A

NAMESPACE   NAME                 READY   SECRET               AGE
my-svc-ns   letsencrypt-secret   False   letsencrypt-secret   17h

Now it looks like its an issue with the certificate, not the app gateway. I jump over to this documentation: https://cert-manager.io/docs/faq/troubleshooting/ and run kubectl describe certificate letsencrypt-secret -n my-svc-ns

Name:         letsencrypt-secret
Namespace:    my-svc-ns
... boring stuff here ...
Status:
    Message:               The certificate request has failed to complete and will be retried: Failed to wait for order resource "letsencrypt-secret-cxg5p-2149327961" to become ready: order is in "invalid" state:
    Reason:                Failed
    Status:                False
    Type:                  Issuing
    Message:               Issuing certificate as Secret does not exist
    Reason:                DoesNotExist
    Status:                False
    Type:                  Ready
  Last Failure Time:       2020-10-15T07:01:05Z
Events:
  Type     Reason     Age                 From          Message
  ----     ------     ----                ----          -------
  Normal   Issuing    56m (x18 over 17h)  cert-manager  Issuing certificate as Secret does not exist
  Normal   Generated  56m                 cert-manager  Stored new private key in temporary Secret resource "letsencrypt-secret-wcjnp"
  Warning  Failed     56m                 cert-manager  The certificate request has failed to complete and will be retried: Failed to wait for order resource "letsencrypt-secret-cxg5p-2149327961" to become ready: order is in "invalid" state:
  Normal   Requested  56m                 cert-manager  Created new CertificateRequest resource "letsencrypt-secret-lrzgf"

Next up, it looks like the order has failed kubectl describe certificaterequest letsencrypt-secret-x8csq -n my-svc-ns

Name:         letsencrypt-secret-x8csq
Namespace:    my-svc-ns
... more stuff here ...
Status:
  Conditions:
    Last Transition Time:  2020-10-15T08:01:05Z
    Message:               Failed to wait for order resource "letsencrypt-secret-x8csq-2149327961" to become ready: order is in "invalid" state:
    Reason:                Failed
    Status:                False
    Type:                  Ready
  Failure Time:            2020-10-15T08:02:02Z
Events:
  Type     Reason        Age    From          Message
  ----     ------        ----   ----          -------
  Normal   OrderCreated  5m30s  cert-manager  Created Order resource my-svc-ns/letsencrypt-secret-x8csq-2149327961
  Warning  OrderFailed   4m33s  cert-manager  Failed to wait for order resource "letsencrypt-secret-x8csq-2149327961" to become ready: order is in "invalid" state:

Again, you can check the order via kubectl describe orders -A

Name:         letsencrypt-secret-x8csq-2149327961
Namespace:    my-svc-ns
... more of the boring stuff ...
Status:
  Authorizations:
    Challenges:
      Token:        <REDACTED>
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/<REDACTED>
      Token:        <REDACTED>
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/<REDACTED>
      Token:        <REDACTED>
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/<REDACTED>
    Identifier:     example.com
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/<REDACTED>
    Wildcard:       false
  Failure Time:     2020-10-15T08:02:02Z
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/<REDACTED>
  State:            invalid
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/<REDACTED>
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  10m   cert-manager  Created Challenge resource "letsencrypt-secret-x8csq-<redacted>" for domain "example.com"

Looks like the challenge is in an invalid state. Down the rabbit hole we go!! I looked through the service logs and found that the challenge is responding with some html, rather than the actual response.

Here I am using stern to extract the service logs in one go:

stern -n cert-manager cert-manager-webhook -s 2h >> /c/Downloads/cert-manager-webhook-logs.txt
stern -n cert-manager cert-manager -s 2h >> /c/Downloads/cert-manager-logs.txt

I have gone though the logs. Here the the relevant bits:

cert-manager-webhook logs

cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 12:01:05.598962       1 logs.go:58] http: TLS handshake error from 10.2.0.39:56436: EOF
cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 12:01:05.601111       1 logs.go:58] http: TLS handshake error from 10.2.0.39:56438: EOF
cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 12:01:05.613115       1 logs.go:58] http: TLS handshake error from 10.2.0.39:56442: EOF
cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 13:01:05.614956       1 logs.go:58] http: TLS handshake error from 10.2.0.39:33458: EOF

cert-manager logs


12:01:05.598962       1 logs.go:58] http: TLS handshake error from 10.2.0.39:56436: EOF
... this message repeated ~ 5 times ...

11:30:51.601008 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled" ... variations on this message for a few hundred lines ... 11:30:51.601036 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled"

12:01:05.000769 1 conditions.go:162] Found status change for Certificate "letsencrypt-secret" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2020-10-15 12:01:05.000756269 +0000 UTC m=+7349.678490276 13:19:32.480187 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled" "controller"="apiservice" "name"="v1beta1.external.metrics.k8s.io" "namespace"="" "reconcilerGroup"="apiregistration.k8s.io" "reconcilerKind"="APIService" ... variations on this message for a few dozen lines ... 13:19:32.480368 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled" 12:01:05.456313 1 conditions.go:162] Found status change for Certificate "letsencrypt-secret" condition "Issuing": "True" -> "False";

12:01:05.470695 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474184 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474233 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474254 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474313 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"

12:01:05.477599 1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="my-svc-ns/letsencrypt-secret" "retry_after"="2020-10-15T13:01:05Z" 12:01:05.488750 1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found" "related_resource_kind"="CertificateRequest" 12:01:05.488872 1 controller.go:142] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961\" not found"

12:01:05.519805 1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961\" not found" 12:01:05.589204 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"letsencrypt-secret\": the object has been modified; please apply your changes to the latest version and try again" "key"="my-svc-ns/letsencrypt-secret"

12:01:05.621883 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "letsencrypt-secret-dbgxb" condition "Ready" to 2020-10-15 12:01:05.621874877 +0000 UTC m=+7350.299608884 12:01:05.695702 1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961\" not found" 12:01:05.696288 1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961-1861071662\" not found"

12:01:09.892470 1 pod.go:70] cert-manager/controller/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" 12:01:09.934564 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" 12:01:09.935267 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" 12:01:09.935640 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" 12:01:10.023099 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="did not get expected response when querying endpoint, expected \"\" but got: <!DOCTYPE html>\n<html la... (truncated)" ... this message repeated ~ 10 times ...

12:02:00.229824 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" 12:02:00.229930 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" 12:02:00.230006 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" 12:02:00.242971 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '502', expected '200'" ... this message repeated ~ 10 times ...

12:03:02.337775 1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for example.com: 403 urn:ietf:params:acme:error:caa: CAA record for example.com prevents issuance"


The following error stands out:

acme: authorization error for example.com: 403 urn:ietf:params:acme:error:caa: CAA record for example.com prevents issuance


But this is likely due to the previous 502 and error pages:

wrong status code '502', expected '200' did not get expected response when querying endpoint, expected \"\" but got: <!DOCTYPE html>\n<html la... (truncated)"



Frustratingly, the HTML is truncated so I am unable to fully see if it is my webapp (served via ingress route /*) or if its the good old fashioned 404 / 502 page that app gateway produces.
Something fishy is going on though, I followed the azure documentation almost exactly and it seems like the gateway itself is passing back a 502.

==================

I appreciate that the cert-manager app isn't an Azure product, but getting an SSL cert for your ingress is a vital step that most people using the gateway will need to do!
It seems to me that some extra step is missing from the instructions here. Or maybe my configuration is wrong?

@akshaysngupta ?

akshaysngupta commented 3 years ago

@thatguycalledrob thanks a lot for such a detailed post. Let me reproduce this and post back.

moaxaca commented 3 years ago

Facing a similar issue. This is a major blocker to adoption.

thatguycalledrob commented 3 years ago

@akshaysngupta - Wanted to give you an update.

My problem was incorrectly set CAA records. To sys admins, proper setup of the CAA record should be an obvious step, but in my case as a backend-dev-doing-devops it wasn't.

For my organisation our CAA records are set via ticket to a central DNS team (sigh), and its a whitelist with no default values. After reading the cert-bot docs for another project, I realised that I hadn't checked the CAA record. One IT ticket later and my above issue went away!

Since the error logs are pretty rubbish in this case, it may be worth adding some comment around CAA record setup for the uninitiated!

chenlonglab198 commented 2 years ago

Facing the same issue here. May I know anyone got a resolution already? or it's still the same, like a bug?

Azure / application-gateway-kubernetes-ingress

Comprehensive Reference/Documentation #758