Rewrite-target and wildcard path rule problem

mleneveut commented 3 years ago

Describe the bug Following the ingress doc, I'm trying to create an ingress with URL /api/xxx being forwarded to the pod with just /xxx https://docs.microsoft.com/fr-fr/azure/aks/ingress-basic

So I added a nginx.ingress.kubernetes.io/rewrite-target: /$2 and a pathRule containing /api(/|$)(.*)

I get an error :

network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /api(/|$)(.*) in the path rule /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/applicationGateways/xxx/urlPathMaps/url-f3ab40d69199895b8de934c7e1d73a5d/pathRules/pr-develop-bff-ingress-rule-0-path-0 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '_', '~', '!', '$', '(' , ')', '*', '\\'', '+', ',', ';', '=', ':', '@']" Details=[]

To Reproduce Try to create the following ingress :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bff-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/connection-draining: "true"
    appgw.ingress.kubernetes.io/connection-draining-timeout: "30"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
      - contoso.com
    secretName: tls-secret
  rules:
  - host: contoso.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: service
            port:
              number: 80

Ingress Controller details

kubectl describe pod ingress-azure-589f6d6585-7jglz
Name:         ingress-azure-589f6d6585-7jglz
Namespace:    default
Priority:     0
Node:         aks-nodepool1-14107490-0/10.0.0.105
Start Time:   Wed, 17 Feb 2021 16:24:15 +0100
Labels:       app=ingress-azure
              pod-template-hash=589f6d6585
              release=ingress-azure
Annotations:  checksum/config: 7e279491deac64d3ece4f59f54a11f67d4311a3df239599dfbf4a58d5e1c178f
              prometheus.io/port: 8123
              prometheus.io/scrape: true
Status:       Running
IP:           10.0.0.142
IPs:
  IP:           10.0.0.142
Controlled By:  ReplicaSet/ingress-azure-589f6d6585
Containers:
  ingress-azure:
    Container ID:   containerd://01d08190486ece7b97ef9f628afc97c16bdc83c13b5b8d7eb6304cd58782ac2d
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.3.0
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:28cbb5581b775523821742119f57b25dd584697b6c1c05c5ddeabf2fb59f37c7
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 17 Feb 2021 16:24:30 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  ingress-azure-589f6d6585-7jglz (v1:metadata.name)
      AGIC_POD_NAMESPACE:             default (v1:metadata.namespace)
      AZURE_AUTH_LOCATION:            /etc/Azure/Networking-AppGW/auth/armAuth.json
    Mounts:
      /etc/Azure/Networking-AppGW/auth from networking-appgw-k8s-azure-service-principal-mount (ro)
      /etc/appgw/ from azure (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-token-n2k8k (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  Directory
  networking-appgw-k8s-azure-service-principal-mount:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  networking-appgw-k8s-azure-service-principal
    Optional:    false
  ingress-azure-token-n2k8k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-azure-token-n2k8k
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                     Age                From                                                       Message
  ----     ------                     ----               ----                                                       -------
  Normal   Scheduled                  <unknown>                                                                     Successfully assigned default/ingress-azure-589f6d6585-7jglz to aks-nodepool1-14107490-0
  Normal   Pulling                    15m                kubelet, aks-nodepool1-14107490-0                          Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.3.0"
  Normal   Pulled                     15m                kubelet, aks-nodepool1-14107490-0                          Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.3.0" in 8.236957173s
  Normal   Created                    15m                kubelet, aks-nodepool1-14107490-0                          Created container ingress-azure
  Normal   Started                    15m                kubelet, aks-nodepool1-14107490-0                          Started container ingress-azure
  Warning  FailedApplyingAppGwConfig  31s (x4 over 15m)  azure/application-gateway, ingress-azure-589f6d6585-7jglz  network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /api(/|$)(.*) in the path rule /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/applicationGateways/xxx/urlPathMaps/url-f3ab40d69199895b8de934c7e1d73a5d/pathRules/pr-develop-bff-ingress-rule-0-path-0 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '_', '~', '!', '$', '(' , ')', '*', '\\'', '+', ',', ';', '=', ':', '@']" Details=[]

Output of `kubectl logs .

kubectl logs ingress-azure-589f6d6585-7jglz
ERROR: logging before flag.Parse: I0217 15:24:30.144440       1 utils.go:115] Using verbosity level 1 from environment variable APPGW_VERBOSITY_LEVEL
I0217 15:24:30.259668       1 environment.go:246] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0217 15:24:30.259997       1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json
I0217 15:24:30.260130       1 httpserver.go:57] Starting API Server on :8123
I0217 15:24:30.675885       1 main.go:175] Ingress Controller will observe all namespaces.
I0217 15:24:30.853118       1 context.go:138] k8s context run started
I0217 15:24:30.853182       1 context.go:188] Waiting for initial cache sync
I0217 15:24:31.153504       1 context.go:201] Initial cache sync done
I0217 15:24:31.153841       1 context.go:202] k8s context run finished
I0217 15:24:31.154112       1 worker.go:39] Worker started
E0217 15:24:31.673262       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /api(/|$)(.*) in the path rule /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/applicationGateways/xxx/urlPathMaps/url-f3ab40d69199895b8de934c7e1d73a5d/pathRules/pr-develop-bff-ingress-rule-0-path-0 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '_', '~', '!', '$', '(' , ')', '*', '\\'', '+', ',', ';', '=', ':', '@']" Details=[]
E0217 15:24:31.673444       1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /api(/|$)(.*) in the path rule /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/applicationGateways/xxx/urlPathMaps/url-f3ab40d69199895b8de934c7e1d73a5d/pathRules/pr-develop-bff-ingress-rule-0-path-0 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '_', '~', '!', '$', '(' , ')', '*', '\\'', '+', ',', ';', '=', ':', '@']" Details=[]

Any Azure support tickets associated with this issue.

mleneveut commented 3 years ago

I fixed the deployment problem with this ingress code :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bff-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/connection-draining: "true"
    appgw.ingress.kubernetes.io/connection-draining-timeout: "30"
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  tls:
  - hosts:
      - contoso.com
    secretName: tls-secret
  rules:
  - host: contoso.com
    http:
      paths:
      - path: /my-url-context/api/*
        pathType: Prefix
        backend:
          service:
            name: service
            port:
              number: 80

but not sure it does what I want. I want to access the service with an URL like contoso.com/my-url-context/api/my-endpoint and that the service sees/receives an URL /my-endpoint

bhavya11 commented 3 years ago

Hello mleneveut, Please remove the following annotation: appgw.ingress.kubernetes.io/backend-path-prefix: "/" It should resolve your issue.

ClumsyPotato commented 3 years ago

Hey i have the same issue. My ingress file looks like this

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  creationTimestamp: null
  name: appgwapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - path: /pentests/appgw/*
        pathType: Prefix
        backend:
          service:
            name: appgwapp
            port:
              number: 8080

My app listens on /hello and /hello/buddy and i want to access it through my gateway through /pentests/appgw/hello or /pentests/appgw/hello/buddy. I tested it and its not working. Any idea why?

rsun-thoughtworks commented 2 years ago

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: appgwapp annotations: kubernetes.io/ingress.class: azure/application-gateway spec: rules:

http: paths:

path: /pentests/appgw/* pathType: Prefix backend: service: name: appgwapp port: number: 8080

You can try this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  creationTimestamp: null
  name: appgwapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  rules:
  - http:
      paths:
      - path: /pentests/appgw/*
        pathType: Prefix
        backend:
          service:
            name: appgwapp
            port:
              number: 8080

This should work for you.

Also remember that you must have readinessProbe and livenessProbe configured in your deployments.

Azure / application-gateway-kubernetes-ingress

Rewrite-target and wildcard path rule problem #1145