Can't create AWX with LoadBalancer in EKS

[iuvooneill]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

I could be daft and doing something wrong, but I am attempting to use the demo instructions but with an EKS environment. I'm using "service_type: LoadBalancer", but no load balancer is getting created.

I've tried this with 0.28.0, and I updated to 0.30.0, with no difference.

My awx-demo.yaml: --- apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx-demo spec: service_type: LoadBalancer loadbalancer_port: 8443 loadbalancer_protocol: https

I have also tried with the following for the EKS environment to use an ALB: spec: ... ingress_type: ingress hostname: awx-demo.domain.com ingress_path: "/*" ingress_path_type: Prefix ingress_annotations: | kubernetes.io/ingress.class: "alb" alb.ingress.kubernetes.io/scheme: "internet-facing" alb.ingress.kubernetes.io/target-type: "ip" alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'

Making these changes does not seem to result in anything actually changing in the deployment.

AWX Operator version

0.30.0

AWX version

latest (default)

Kubernetes platform

other (please specify in additional information)

Kubernetes/Platform version

1.23

Modifications

no

Steps to reproduce

Set up EKS Environment

In a directory with kustomization.yaml and awx-demo.yaml, run either kubectl apply -k . or kustomize build . | kubectl apply -f - awx-demo.yaml.txt kustomization.yaml.txt

Expected results

AWX deployed with a load balancer of some form.

Actual results

Output is: namespace/awx created customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created serviceaccount/awx-operator-controller-manager created role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created role.rbac.authorization.k8s.io/awx-operator-leader-election-role created clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created configmap/awx-operator-awx-manager-config created service/awx-operator-controller-manager-metrics-service created deployment.apps/awx-operator-controller-manager created awx.awx.ansible.com/awx-demo created

But not load balancer is created. Services show as: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE awx-demo-postgres-13 ClusterIP None <none> 5432/TCP 10m awx-operator-controller-manager-metrics-service ClusterIP 10.100.233.67 <none> 8443/TCP 11m

Additional information

AWS EKS is used for environment

Operator Logs

No response

[JustOneMoreDog]

Installing Amazon Load Balancer Controller

• Next up we need to install the load balancer controller. Whenever we make an ingress resource, EKS will automatically create an application load balancer for us
• We will be following the instructions found in these links
  • https://aws.amazon.com/blogs/containers/introducing-aws-load-balancer-controller/
  • https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html
  • https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

Follow those guides and you will be able to get it setup. Also, do not use service_type: LoadBalancer

here is my awx-deployment.yaml file

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: ClusterIP
  hostname: awx.example.com
  ingress_type: ingress
  ingress_annotations: |
    alb.ingress.kubernetes.io/scheme: internal
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/group.name: "poc-dev"
    external-dns.alpha.kubernetes.io/hostname: awx-ssl3.example.com
    alb.ingress.kubernetes.io/tags: Environment=dev

[iuvooneill]

I must still be missing something. I've created the policy, RBAC role, service account, and deployed the controller, but I get in the logs:

main.go:97] no matches for kind "Ingress" in version "extensions/v1beta1"

and that seems to be before I've tried to deploy AWX.

[vrunda87]

@JustOneMoreDog & @iuvooneill - Question (if you are running AWX in load balancer setup) how have you configured your execution environments? Can we make them dynamic?

[jbelmont]

@JustOneMoreDog I have tried to create an ALB as well using the following:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - github.com/ansible/awx-operator/config/default?ref=2.5.3
  - awx.yaml

# Set the image tags to match the git version from above
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.5.3

# Specify a custom namespace in which to install AWX
namespace: awx

awx.yaml

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  admin_user: admin
  admin_password_secret: some-secret-thinger
  service_type: ClusterIP
  loadbalancer_port: 443
  loadbalancer_protocol: https
  ingress_type: ingress
  hostname: awx.<our_hosted_name>.com
  ingress_path: /
  ingress_path_type: Prefix
  ingress_annotations: |
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: "internal"
    alb.ingress.kubernetes.io/certificate-arn: "<REDACTED_ARN"
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/load-balancer-attributes: "routing.http.x_amzn_tls_version_and_cipher_suite.enabled=true"
    alb.ingress.kubernetes.io/tags: "Name=<OUR_NAME>"
...

I see a controller manager installed, local postgres created several tasks and then a web ui pod is created as well. No load balancer is ever created, I did try using the service_type: LoadBalancer directive but that creates a Classic Load Balancer as others mentioned.

I made sure to follow the docs you listed in the earlier comment thread and see:

kubectl get deployment -n kube-system aws-load-balancer-controller
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           361d

Everything is setup as the docs suggest from what I can tell. Additionally I saw errors in the web ui pod and read an issue where a solution was proposed using the awx-manage cli and exec'ed into the web ui container and ran:

awx-manage migrate --noinput

This command finally helps the migration that is supposed to finish resolve and get the pod running healthy. I am not able to get DNS resolution as I can't get an Application Load balancer created and point Route 53 to what is supposed to be the created ALB. The docs are not helpful in that the complete picture on how to get this running on an existing eks cluster with alb isn't really explained and I ended reading an outdated blog post from https://briangaber.hashnode.dev/how-to-install-awx-2010-on-aws-eks#install-awx-operator-and-deploy-awx which at least pointed me to apiVersion: awx.ansible.com/v1beta1. I have tried a whole bunch of options by downgrading to 0.20.0, upgrading to 0.30.0. I have gone from 1.0.0 to the latest 2.5.3 and have even tried the latest devel branch nothing works to create the ALB. The only thing that actually creates a LoadBalancer is the service_type: LoadBalancer directive but that is a Classic LoadBalancer which isn't what is wanted.