Marketplace nginx Ingress controller not working with a valid Ingress resource

[omkensey]

This issue is a:

• [X] Bug Report

If the issue concerns an existing Marketplace application, please tag the maintainer of the application (found in manifest.yaml) below: @saiyam1814

• Expected behaviour Ingress resource created by a Helm chart works
• Encountered behaviour Ingress resource did not work
• The steps to recreate the behaviour that you are reporting I initially created a cluster with the Nginx Ingress controller installed from the marketplace. I wanted to run a newer version of Vault than what the Marketplace app installs, so I installed it via Helm directly, with some values set so that I could run an HA cluster with an ingress:

  helm upgrade --install --atomic vault \
  --set server.ingress.enabled=true \
  --set server.ha.enabled=true \
  --set server.ha.raft.enabled=true \
  --set server.ingress.hosts[0].host=[my cluster's DNS name] \
  --set server.ingress.hosts[0].paths[0]='/vault(/|$)(.*)' --set server.ingress.annotations='nginx.ingress.kubernetes.io/rewrite-target: /$2' \
  --set server.ingress.ingressClassName="nginx" \
  --set server.ingress.pathType=Prefix \
  hashicorp/vault

  This created the following Ingress:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
  annotations:
  meta.helm.sh/release-name: vault
  meta.helm.sh/release-namespace: default
  nginx.ingress.kubernetes.io/rewrite-target: /$2
  creationTimestamp: "2021-11-22T02:29:23Z"
  generation: 1
  labels:
  app.kubernetes.io/instance: vault
  app.kubernetes.io/managed-by: Helm
  app.kubernetes.io/name: vault
  helm.sh/chart: vault-0.18.0
  name: vault
  namespace: default
  resourceVersion: "1635"
  uid: 616580b3-cd1b-4d0a-9bc1-fdb97dbe6a2d
  spec:
  ingressClassName: nginx
  rules:
  - host: [my cluster's DNS name]
  http:
    paths:
    - backend:
        service:
          name: vault-active
          port:
            number: 8200
      path: /vault(/|$)(.*)
      pathType: Prefix
  status:
  loadBalancer:
  ingress:
  - ip: [my cluster's LoadBalancer IP]

Using the marketplace version of the Kubernetes nginx ingress, this Ingress did nothing -- I saw in the controller logs that it was accepted but no traffic ever was logged and all access attempts gave me a 404 from nginx. When I spun up a new cluster without the marketplace nginx Ingress controller installed, and installed it manually using the Helm chart provided by the project, the above ingress worked as expected and after unsealing, I was able to set VAULT_ADDR in my client to http://[my cluster DNS address]/vault and use the Vault client to manage the cluster as normal.

[saiyam1814]

@dirsigler is there something we can do to fix this?

[dirsigler]

Has probably to do with the recent changes in the Ingress API and the Kubernetes Version of Civo moving forward. This will probably affect also other Users with other Applications as the upstream Helm Charts need to support the newest Ingress syntax.

Will update and test the Vault version provided in the Civo Marketplace which should hopefully fix this issue.

[dirsigler]

First I thought I miss-read what @omkensey wrote, because he stated that there is a problem with the Nginx application provided in the marketplace and not with the Vault application. But after testing now several things I think it is not a problem with Vault and maybe not a problem with NGINX.

In my tests there are problems with the --set server.ingress.hosts flags and the annotation. I was not able to create a working Ingress resource like that, neither with the Civo Marketplace Nginx Ingress nor with the manual Helm installed ingress-nginx version 4.0.9.

What worked with both Ingress(es?) was using a subdomain in front of the Civo DNS instead. Means my resulting Ingress resource looks something like:

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      meta.helm.sh/release-name: vault
      meta.helm.sh/release-namespace: vault                        # remove the rewrite annotation
    creationTimestamp: "2021-11-22T10:04:28Z"
    generation: 2
    labels:
      app.kubernetes.io/instance: vault
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: vault
      helm.sh/chart: vault-0.18.0
    name: vault
    namespace: vault
    resourceVersion: "1472"
    uid: bb3a9985-fcef-4337-a858-389aadf400c8
  spec:
    ingressClassName: nginx
    rules:
    - host: vault.my-unique-dns-string.k8s.civo.com                 # add a just a subdomain to the Civo DNS
      http:
        paths:
        - backend:
            service:
              name: vault-active
              port:
                number: 8200
          path: /                                                      # changed the path
          pathType: Prefix
  status:
    loadBalancer:
      ingress:
      - ip: 74.220.29.204
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Maybe I miss something and there is a difference between the Nginx Ingress resources which are installed via the Marketplace and the ones installed via the official Helm Chart way, but I could not make it work with either in the way @omkensey specified.

[dirsigler]

Even tho I think this issue is more related to either a misconfiguration or a problem with the marketplace provided Nginx Ingress I created a PR to (finally) update Vault. https://github.com/civo/kubernetes-marketplace/pull/279

[omkensey]

With my Ingress resource, rather than host-based routing I was using path-based routing. That was why I needed the rewrite-rule annotation and the capture groups in the path. I get a warning about maps vs. strings on the annotations when I do the Helm install, but it works (gives the correct annotation in the manifest).

The other thing I noticed about the Marketplace version vs the version installed directly from the Helm chart is it looked like the Marketplace version created two Jobs that looked like they did something with admission controllers, but the Helm-installed version didn't. (I don't know why this is since looking at the chart it looks like the Helm chart should create those jobs by default.)

Also:

Even tho I think this issue is more related to either a misconfiguration or a problem with the marketplace provided Nginx Ingress I created a PR to (finally) update Vault.

I appreciate it, I was going to look at that next :)

[ionut-maxim]

I also ran tests and as @dirsigler said, at this moment with ingress-nginx (either marketplace or official helm chart) I am unable to used domains other than the one assigned to the cluster by Civo.

[ionut-maxim]

I also ran tests and as @dirsigler said, at this moment with ingress-nginx (either marketplace or official helm chart) I am unable to used domains other than the one assigned to the cluster by Civo.

This was actually an issue on my side with misconfigured DNS.