Open omkensey opened 2 years ago
@dirsigler is there something we can do to fix this?
Has probably to do with the recent changes in the Ingress API and the Kubernetes Version of Civo moving forward. This will probably affect also other Users with other Applications as the upstream Helm Charts need to support the newest Ingress syntax.
Will update and test the Vault version provided in the Civo Marketplace which should hopefully fix this issue.
First I thought I miss-read what @omkensey wrote, because he stated that there is a problem with the Nginx application provided in the marketplace and not with the Vault application. But after testing now several things I think it is not a problem with Vault and maybe not a problem with NGINX.
In my tests there are problems with the --set server.ingress.hosts
flags and the annotation. I was not able to create a working Ingress resource like that, neither with the Civo Marketplace Nginx Ingress nor with the manual Helm installed ingress-nginx version 4.0.9.
What worked with both Ingress(es?) was using a subdomain in front of the Civo DNS instead. Means my resulting Ingress resource looks something like:
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
meta.helm.sh/release-name: vault
meta.helm.sh/release-namespace: vault # remove the rewrite annotation
creationTimestamp: "2021-11-22T10:04:28Z"
generation: 2
labels:
app.kubernetes.io/instance: vault
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: vault
helm.sh/chart: vault-0.18.0
name: vault
namespace: vault
resourceVersion: "1472"
uid: bb3a9985-fcef-4337-a858-389aadf400c8
spec:
ingressClassName: nginx
rules:
- host: vault.my-unique-dns-string.k8s.civo.com # add a just a subdomain to the Civo DNS
http:
paths:
- backend:
service:
name: vault-active
port:
number: 8200
path: / # changed the path
pathType: Prefix
status:
loadBalancer:
ingress:
- ip: 74.220.29.204
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Maybe I miss something and there is a difference between the Nginx Ingress resources which are installed via the Marketplace and the ones installed via the official Helm Chart way, but I could not make it work with either in the way @omkensey specified.
Even tho I think this issue is more related to either a misconfiguration or a problem with the marketplace provided Nginx Ingress I created a PR to (finally) update Vault. https://github.com/civo/kubernetes-marketplace/pull/279
With my Ingress resource, rather than host-based routing I was using path-based routing. That was why I needed the rewrite-rule annotation and the capture groups in the path. I get a warning about maps vs. strings on the annotations when I do the Helm install, but it works (gives the correct annotation in the manifest).
The other thing I noticed about the Marketplace version vs the version installed directly from the Helm chart is it looked like the Marketplace version created two Jobs that looked like they did something with admission controllers, but the Helm-installed version didn't. (I don't know why this is since looking at the chart it looks like the Helm chart should create those jobs by default.)
Also:
Even tho I think this issue is more related to either a misconfiguration or a problem with the marketplace provided Nginx Ingress I created a PR to (finally) update Vault.
I appreciate it, I was going to look at that next :)
I also ran tests and as @dirsigler said, at this moment with ingress-nginx
(either marketplace or official helm chart) I am unable to used domains other than the one assigned to the cluster by Civo.
I also ran tests and as @dirsigler said, at this moment with
ingress-nginx
(either marketplace or official helm chart) I am unable to used domains other than the one assigned to the cluster by Civo.
This was actually an issue on my side with misconfigured DNS.
This issue is a:
If the issue concerns an existing Marketplace application, please tag the maintainer of the application (found in
manifest.yaml
) below: @saiyam1814This created the following Ingress:
Using the marketplace version of the Kubernetes nginx ingress, this Ingress did nothing -- I saw in the controller logs that it was accepted but no traffic ever was logged and all access attempts gave me a 404 from nginx. When I spun up a new cluster without the marketplace nginx Ingress controller installed, and installed it manually using the Helm chart provided by the project, the above ingress worked as expected and after unsealing, I was able to set VAULT_ADDR in my client to
http://[my cluster DNS address]/vault
and use the Vault client to manage the cluster as normal.