ingress_path needs a * for GKE ingress

sdigit commented 2 years ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

On GKE, I use an HTTP(s) load balancer. In order for it to work, I needed to modify the ingress spec the operator creates to append a /* to it. See Additional information for the details.

I'm happy to work on putting this into a proper PR, my question is how to do it - I understand that Google-specific settings are probably not desirable, does anyone have a suggestion?

AWX Operator version

0.22.0

AWX version

21.0.0

Kubernetes platform

other (please specify in additional information)

Kubernetes/Platform version

v1.21.11-gke.900

Modifications

yes

Steps to reproduce

Deploy AWX in GKE with an HTTP load balancer in front of it Deployment will succeed, but AWX will never become usable as it won't be routing the correct path prefix.

Expected results

AWX to come up and work

Actual results

I needed to modify the operator to append a /*

Additional information

Spec snippet:

spec:
  ingress_type: ingress
  ingress_path: /
  ingress_path_type: ImplementationSpecific
  ingress_tls_secret: awx-domain-com-tls
  ingress_annotations: |
    kubernetes.io/ingress.global-static-ip-name: awx-lb
    networking.gke.io/v1beta1.FrontendConfig: "tls-modern"
    cloud.google.com/backend-config: '{"ports": {"80":"awx-backendconfig"}}'
  service_annotations: |
    cloud.google.com/backend-config: '{"ports": {"80":"awx-backendconfig"}}'
    cloud.google.com/neg: '{"ingress": true}'

The patch I created:

diff --git a/roles/installer/templates/ingress.yaml.j2 b/roles/installer/templates/ingress.yaml.j2
index 57cf42e..715ffb2 100644
--- a/roles/installer/templates/ingress.yaml.j2
+++ b/roles/installer/templates/ingress.yaml.j2
@@ -19,7 +19,7 @@ spec:
   rules:
     - http:
         paths:
-          - path: '{{ ingress_path }}'
+          - path: '{{ (ingress_path + '/*').replace("//","/") }}'
             pathType: '{{ ingress_path_type }}'
             backend:
               service:

The final ingress looks like this:

spec:
  rules:
  - host: awx.domain.com
    http:
      paths:
      - backend:
          service:
            name: awx-service
            port:
              number: 80
        path: /*
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - awx.domain.com
    secretName: awx-domain-com-tls
status:
  loadBalancer:
    ingress:
    - ip: 1.2.3.4

Operator Logs

No response

sdigit commented 2 years ago

Without the patch, I tried to set ingress_path to /* but that broke the generated nginx config.

john-westcott-iv commented 2 years ago

@rooftopcellist Do we have any kind of "platform" tags that we could use for a Jinja if condition in this template?

rooftopcellist commented 2 years ago

We don't have any "platform" tags or variables to key off of, and I think we should avoid adding too much platform specific logic to the awx-operator if possible.

After chatting on IRC the idea of adding an ingress_path_suffix variable would be a good approach that would solve this problem, while still keeping the roles generic.

This variable should also be configurable via the spec.

lennarthaller commented 2 years ago

Hey @sdigit,

Thank you for your already quite thorough post. I am facing the same issue, but my understanding of Kubernetes so far is limited so I haven't been able to resolve this. I would greatly appreciate some guidance.

What I have done so far:

Fork the repo
Apply your patch
Build a new image
Change the tag and git url in the kustomization.yaml to my fork and create a awx.yaml:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - <my-repo-url>ref=<my-tag>
  - awx-deployment.yaml
images:
  - name: <my-repo-url>
    newTag: <my-tag>

awx-deployment.yaml:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-deployment
spec:
  ingress_annotations: |
    kubernetes.io/ingress.global-static-ip-name: awx-ip
    cloud.google.com/backend-config: '{"ports": {"80":"awx-backendconfig"}}'
  service_annotations: |
    cloud.google.com/backend-config: '{"ports": {"80":"awx-backendconfig"}}'
    cloud.google.com/neg: '{"ingress": true}'

I removed the tls line as I want to get it running over http first and then move to https once it works.

Result is a 502 and Kubernetes proclaims that the "Backend is unhealthy".

Any steps I am missing to apply your fix and deploy AWX on GKE?

Thank you so much for your time!

sdigit commented 2 years ago

Any steps I am missing to apply your fix and deploy AWX on GKE?

I just had to redo this fix myself for awx-operator 0.28.0; here's exactly what I did (in a checkout of awx-operator 0.28.0):

edit roles/installer/templates/networking/ingress.yaml.j2 and make the necessary modification
run docker build -t <my repository>/awx-operator:0.28.0-modified .
run docker push <my repository>/awx-operator:0.28.0-modified
run make gen-resources | tee /tmp/operator.yaml
edit /tmp/operator.yaml to change the image: reference to point to the one I just pushed
deploy /tmp/operator.yaml however you prefer (I use a gitops setup)

ansible / awx-operator