search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.9k stars 3.4k forks source link

K8S Install Fails in AWX 8.0.0 #5222

Closed Quantas closed 4 years ago

Quantas commented 4 years ago
ISSUE TYPE
SUMMARY
ENVIRONMENT
STEPS TO REPRODUCE
  1. Setup a brand new Kubernetes cluster (GKE, Rancher, etc.)
  2. Clone a fresh copy of the 8.0.0 tag
  3. Ensure Helm is installed on cluster according to official documentation
  4. Configure inventory, setting kubernetes_context, kubernetes_namespace, tiller_namespace, leave everything else default
  5. ansible-playbook -i inventory install.yml
  6. The install fails at the task: [kubernetes : Apply Deployment]
EXPECTED RESULTS

Successful installation.

ACTUAL RESULTS
changed: [localhost] => (item=apiVersion: v1
kind: ConfigMap
metadata:
  name: awx-config
  namespace: awx
data:
  awx_settings: |
    import os
    import socket
    ADMINS = ()

    AWX_PROOT_ENABLED = True

    # Automatically deprovision pods that go offline
    AWX_AUTO_DEPROVISION_INSTANCES = True

    SYSTEM_TASK_ABS_CPU = 6
    SYSTEM_TASK_ABS_MEM = 20

    INSIGHTS_URL_BASE = "https://example.org"

    #Autoprovisioning should replace this
    CLUSTER_HOST_ID = socket.gethostname()
    SYSTEM_UUID = '00000000-0000-0000-0000-000000000000'

    SESSION_COOKIE_SECURE = False
    CSRF_COOKIE_SECURE = False

    REMOTE_HOST_HEADERS = ['HTTP_X_FORWARDED_FOR']

    STATIC_ROOT = '/var/lib/awx/public/static'
    PROJECTS_ROOT = '/var/lib/awx/projects'
    JOBOUTPUT_ROOT = '/var/lib/awx/job_status'
    SECRET_KEY = open('/etc/tower/SECRET_KEY', 'rb').read().strip()
    ALLOWED_HOSTS = ['*']
    INTERNAL_API_URL = 'http://127.0.0.1:8052'
    SERVER_EMAIL = 'root@localhost'
    DEFAULT_FROM_EMAIL = 'webmaster@localhost'
    EMAIL_SUBJECT_PREFIX = '[AWX] '
    EMAIL_HOST = 'localhost'
    EMAIL_PORT = 25
    EMAIL_HOST_USER = ''
    EMAIL_HOST_PASSWORD = ''
    EMAIL_USE_TLS = False

    LOGGING['handlers']['console'] = {
        '()': 'logging.StreamHandler',
        'level': 'DEBUG',
        'formatter': 'simple',
    }

    LOGGING['loggers']['django.request']['handlers'] = ['console']
    LOGGING['loggers']['rest_framework.request']['handlers'] = ['console']
    LOGGING['loggers']['awx']['handlers'] = ['console']
    LOGGING['loggers']['awx.main.commands.run_callback_receiver']['handlers'] = ['console']
    LOGGING['loggers']['awx.main.commands.inventory_import']['handlers'] = ['console']
    LOGGING['loggers']['awx.main.tasks']['handlers'] = ['console']
    LOGGING['loggers']['awx.main.scheduler']['handlers'] = ['console']
    LOGGING['loggers']['django_auth_ldap']['handlers'] = ['console']
    LOGGING['loggers']['social']['handlers'] = ['console']
    LOGGING['loggers']['system_tracking_migrations']['handlers'] = ['console']
    LOGGING['loggers']['rbac_migrations']['handlers'] = ['console']
    LOGGING['loggers']['awx.isolated.manager.playbooks']['handlers'] = ['console']
    LOGGING['handlers']['callback_receiver'] = {'class': 'logging.NullHandler'}
    LOGGING['handlers']['task_system'] = {'class': 'logging.NullHandler'}
    LOGGING['handlers']['tower_warnings'] = {'class': 'logging.NullHandler'}
    LOGGING['handlers']['rbac_migrations'] = {'class': 'logging.NullHandler'}
    LOGGING['handlers']['system_tracking_migrations'] = {'class': 'logging.NullHandler'}
    LOGGING['handlers']['management_playbooks'] = {'class': 'logging.NullHandler'}

    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
            'LOCATION': '{}:{}'.format("localhost", "11211")
        },
        'ephemeral': {
            'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        },
    }

    USE_X_FORWARDED_PORT = True
)
failed: [localhost] (item=---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: awx
  namespace: awx

---
kind: Service
apiVersion: v1
metadata:
  namespace: awx
  name: rabbitmq
  labels:
    app: awx
    type: LoadBalancer
spec:
  type: NodePort
  ports:
   - name: http
     protocol: TCP
     port: 15672
     targetPort: 15672
   - name: amqp
     protocol: TCP
     port: 5672
     targetPort: 5672
  selector:
    app: awx

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: awx
data:
  enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
  rabbitmq_definitions.json: |
      {
        "users":[{"name": "awx", "password": "awxpass", "tags": "administrator"}],
        "permissions":[
          {"user":"awx","vhost":"awx","configure":".*","write":".*","read":".*"}
        ],
        "vhosts":[{"name":"awx"}],
        "policies":[
            {"vhost":"awx","name":"ha-all","pattern":".*","definition":{"ha-mode":"all","ha-sync-mode":"automatic"}}
        ]
      }
  rabbitmq.conf: |
      ## Clustering
      management.load_definitions = /etc/rabbitmq/rabbitmq_definitions.json
      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc
      cluster_formation.k8s.address_type = ip
      cluster_formation.node_cleanup.interval = 10
      cluster_formation.node_cleanup.only_log_warning = false
      cluster_partition_handling = autoheal
      ## queue master locator
      queue_master_locator=min-masters
      ## enable guest user
      loopback_users.guest = false

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: awx
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: awx
subjects:
- kind: ServiceAccount
  name: awx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: endpoint-reader

---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: awx
  namespace: awx
spec:
  serviceName: awx
  replicas: 1
  template:
    metadata:
      labels:
        name: awx-web-deploy
        service: django
        app: awx
    spec:
      serviceAccountName: awx
      terminationGracePeriodSeconds: 10
      containers:
        - name: awx-web
          image: "ansible/awx_web:8.0.0"
          imagePullPolicy: Always
          ports:
            - containerPort: 8052
          volumeMounts:
            - name: awx-application-config
              mountPath: "/etc/tower/settings.py"
              subPath: settings.py
              readOnly: true

            - name: "awx-application-credentials"
              mountPath: "/etc/tower/conf.d/"
              readOnly: true

            - name: awx-secret-key
              mountPath: "/etc/tower/SECRET_KEY"
              subPath: SECRET_KEY
              readOnly: true

          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"
        - name: awx-celery
          securityContext:
            privileged: true
          image: "ansible/awx_task:8.0.0"
          command:
            - /usr/bin/launch_awx_task.sh
          imagePullPolicy: Always
          volumeMounts:
            - name: awx-application-config
              mountPath: "/etc/tower/settings.py"
              subPath: settings.py
              readOnly: true

            - name: "awx-application-credentials"
              mountPath: "/etc/tower/conf.d/"
              readOnly: true

            - name: awx-secret-key
              mountPath: "/etc/tower/SECRET_KEY"
              subPath: SECRET_KEY
              readOnly: true
          env:
            - name: AWX_SKIP_MIGRATIONS
              value: "1"
          resources:
            requests:
              memory: "2Gi"
              cpu: "1500m"
        - name: awx-rabbit
          image: "ansible/awx_rabbitmq:3.7.4"
          imagePullPolicy: Always
          ports:
            - name: http
              protocol: TCP
              containerPort: 15672
            - name: amqp
              protocol: TCP
              containerPort: 5672
          livenessProbe:
            exec:
              command:
              - /usr/local/bin/healthchecks/rabbit_health_node.py
            initialDelaySeconds: 30
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
              - /usr/local/bin/healthchecks/rabbit_health_node.py
            initialDelaySeconds: 10
            timeoutSeconds: 10
          env:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: RABBITMQ_USE_LONGNAME
              value: "true"
            - name: RABBITMQ_NODENAME
              value: "rabbit@$(MY_POD_IP)"
            - name: RABBITMQ_ERLANG_COOKIE
              valueFrom:
                secretKeyRef:
                  name: "awx-secrets"
                  key: rabbitmq_erlang_cookie
            - name: K8S_SERVICE_NAME
              value: "rabbitmq"
            - name: RABBITMQ_USER
              value: awx
            - name: RABBITMQ_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: "awx-secrets"
                  key: rabbitmq_password
          volumeMounts:
            - name: rabbitmq-config
              mountPath: /etc/rabbitmq
            - name: rabbitmq-healthchecks
              mountPath: /usr/local/bin/healthchecks
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
        - name: awx-memcached
          image: "memcached:latest"
          imagePullPolicy: Always
          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"
      volumes:
        - name: awx-application-config
          configMap:
            name: awx-config
            items:
              - key: awx_settings
                path: settings.py

        - name: "awx-application-credentials"
          secret:
            secretName: "awx-secrets"
            items:
              - key: credentials_py
                path: 'credentials.py'
              - key: environment_sh
                path: 'environment.sh'

        - name: awx-secret-key
          secret:
            secretName: "awx-secrets"
            items:
              - key: secret_key
                path: SECRET_KEY

        - name: rabbitmq-config
          configMap:
            name: rabbitmq-config
            items:
            - key: rabbitmq.conf
              path: rabbitmq.conf
            - key: enabled_plugins
              path: enabled_plugins
            - key: rabbitmq_definitions.json
              path: rabbitmq_definitions.json
        - name: rabbitmq-healthchecks
          configMap:
            name: awx-healthchecks
            items:
            - key: rabbit_health_node.py
              path: rabbit_health_node.py
            defaultMode: 0755
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: awx-healthchecks
  namespace: awx
data:
  rabbit_health_node.py: |
    #!/usr/bin/env python
    try:
      from http.client import HTTPConnection
    except ImportError:
      from httplib import HTTPConnection
    import sys
    import os
    import base64
    authsecret = base64.b64encode(os.getenv('RABBITMQ_USER') + ':' + os.getenv('RABBITMQ_PASSWORD'))
    conn=HTTPConnection('localhost:15672')
    conn.request('GET', '/api/healthchecks/node', headers={'Authorization': 'Basic %s' % authsecret})
    r1 = conn.getresponse()
    if r1.status != 200:
      sys.stderr.write('Received http error %i\n' % (r1.status))
      sys.exit(1)
    body = r1.read()
    if body != '{"status":"ok"}':
      sys.stderr.write('Received body: %s' % body)
      sys.exit(2)
    sys.exit(0)
---
apiVersion: v1
kind: Service
metadata:
  name: awx-web-svc
  namespace: awx
  labels:
    name: awx-web-svc
spec:
  type: "NodePort"
  ports:
    - name: http
      port: 80
      targetPort: 8052
  selector:
    name: awx-web-deploy
---
apiVersion: v1
kind: Service
metadata:
  name: awx-rmq-mgmt
  namespace: awx
  labels:
    name: awx-rmq-mgmt
spec:
  type: ClusterIP
  ports:
    - name: rmqmgmt
      port: 15672
      targetPort: 15672
  selector:
    name: awx-web-deploy
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: awx-web-svc
  namespace: awx
spec:
  backend:
    serviceName: awx-web-svc
    servicePort: 80
) => {"changed": true, "cmd": "echo '---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: awx\n  namespace: awx\n\n---\nkind: Service\napiVersion: v1\nmetadata:\n  namespace: awx\n  name: rabbitmq\n  labels:\n    app: awx\n    type: LoadBalancer\nspec:\n  type: NodePort\n  ports:\n   - name: http\n     protocol: TCP\n     port: 15672\n     targetPort: 15672\n   - name: amqp\n     protocol: TCP\n     port: 5672\n     targetPort: 5672\n  selector:\n    app: awx\n\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: rabbitmq-config\n  namespace: awx\ndata:\n  enabled_plugins: |\n      [rabbitmq_management,rabbitmq_peer_discovery_k8s].\n  rabbitmq_definitions.json: |\n      {\n        \"users\":[{\"name\": \"awx\", \"password\": \"awxpass\", \"tags\": \"administrator\"}],\n        \"permissions\":[\n          {\"user\":\"awx\",\"vhost\":\"awx\",\"configure\":\".*\",\"write\":\".*\",\"read\":\".*\"}\n        ],\n        \"vhosts\":[{\"name\":\"awx\"}],\n        \"policies\":[\n            {\"vhost\":\"awx\",\"name\":\"ha-all\",\"pattern\":\".*\",\"definition\":{\"ha-mode\":\"all\",\"ha-sync-mode\":\"automatic\"}}\n        ]\n      }\n  rabbitmq.conf: |\n      ## Clustering\n      management.load_definitions = /etc/rabbitmq/rabbitmq_definitions.json\n      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s\n      cluster_formation.k8s.host = kubernetes.default.svc\n      cluster_formation.k8s.address_type = ip\n      cluster_formation.node_cleanup.interval = 10\n      cluster_formation.node_cleanup.only_log_warning = false\n      cluster_partition_handling = autoheal\n      ## queue master locator\n      queue_master_locator=min-masters\n      ## enable guest user\n      loopback_users.guest = false\n\n---\nkind: Role\napiVersion: rbac.authorization.k8s.io/v1beta1\nmetadata:\n  name: endpoint-reader\n  namespace: awx\nrules:\n- apiGroups: [\"\"]\n  resources: [\"endpoints\"]\n  verbs: [\"get\"]\n---\nkind: RoleBinding\napiVersion: rbac.authorization.k8s.io/v1beta1\nmetadata:\n  name: endpoint-reader\n  namespace: awx\nsubjects:\n- kind: ServiceAccount\n  name: awx\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: Role\n  name: endpoint-reader\n\n\n---\napiVersion: apps/v1beta1\nkind: StatefulSet\nmetadata:\n  name: awx\n  namespace: awx\nspec:\n  serviceName: awx\n  replicas: 1\n  template:\n    metadata:\n      labels:\n        name: awx-web-deploy\n        service: django\n        app: awx\n    spec:\n      serviceAccountName: awx\n      terminationGracePeriodSeconds: 10\n      containers:\n        - name: awx-web\n          image: \"ansible/awx_web:8.0.0\"\n          imagePullPolicy: Always\n          ports:\n            - containerPort: 8052\n          volumeMounts:\n            - name: awx-application-config\n              mountPath: \"/etc/tower/settings.py\"\n              subPath: settings.py\n              readOnly: true\n\n            - name: \"awx-application-credentials\"\n              mountPath: \"/etc/tower/conf.d/\"\n              readOnly: true\n\n            - name: awx-secret-key\n              mountPath: \"/etc/tower/SECRET_KEY\"\n              subPath: SECRET_KEY\n              readOnly: true\n\n          resources:\n            requests:\n              memory: \"1Gi\"\n              cpu: \"500m\"\n        - name: awx-celery\n          securityContext:\n            privileged: true\n          image: \"ansible/awx_task:8.0.0\"\n          command:\n            - /usr/bin/launch_awx_task.sh\n          imagePullPolicy: Always\n          volumeMounts:\n            - name: awx-application-config\n              mountPath: \"/etc/tower/settings.py\"\n              subPath: settings.py\n              readOnly: true\n\n            - name: \"awx-application-credentials\"\n              mountPath: \"/etc/tower/conf.d/\"\n              readOnly: true\n\n            - name: awx-secret-key\n              mountPath: \"/etc/tower/SECRET_KEY\"\n              subPath: SECRET_KEY\n              readOnly: true\n          env:\n            - name: AWX_SKIP_MIGRATIONS\n              value: \"1\"\n          resources:\n            requests:\n              memory: \"2Gi\"\n              cpu: \"1500m\"\n        - name: awx-rabbit\n          image: \"ansible/awx_rabbitmq:3.7.4\"\n          imagePullPolicy: Always\n          ports:\n            - name: http\n              protocol: TCP\n              containerPort: 15672\n            - name: amqp\n              protocol: TCP\n              containerPort: 5672\n          livenessProbe:\n            exec:\n              command:\n              - /usr/local/bin/healthchecks/rabbit_health_node.py\n            initialDelaySeconds: 30\n            timeoutSeconds: 10\n          readinessProbe:\n            exec:\n              command:\n              - /usr/local/bin/healthchecks/rabbit_health_node.py\n            initialDelaySeconds: 10\n            timeoutSeconds: 10\n          env:\n            - name: MY_POD_IP\n              valueFrom:\n                fieldRef:\n                  fieldPath: status.podIP\n            - name: RABBITMQ_USE_LONGNAME\n              value: \"true\"\n            - name: RABBITMQ_NODENAME\n              value: \"rabbit@$(MY_POD_IP)\"\n            - name: RABBITMQ_ERLANG_COOKIE\n              valueFrom:\n                secretKeyRef:\n                  name: \"awx-secrets\"\n                  key: rabbitmq_erlang_cookie\n            - name: K8S_SERVICE_NAME\n              value: \"rabbitmq\"\n            - name: RABBITMQ_USER\n              value: awx\n            - name: RABBITMQ_PASSWORD\n              valueFrom:\n                secretKeyRef:\n                  name: \"awx-secrets\"\n                  key: rabbitmq_password\n          volumeMounts:\n            - name: rabbitmq-config\n              mountPath: /etc/rabbitmq\n            - name: rabbitmq-healthchecks\n              mountPath: /usr/local/bin/healthchecks\n          resources:\n            requests:\n              memory: \"2Gi\"\n              cpu: \"500m\"\n        - name: awx-memcached\n          image: \"memcached:latest\"\n          imagePullPolicy: Always\n          resources:\n            requests:\n              memory: \"1Gi\"\n              cpu: \"500m\"\n      volumes:\n        - name: awx-application-config\n          configMap:\n            name: awx-config\n            items:\n              - key: awx_settings\n                path: settings.py\n\n        - name: \"awx-application-credentials\"\n          secret:\n            secretName: \"awx-secrets\"\n            items:\n              - key: credentials_py\n                path: '\"'\"'credentials.py'\"'\"'\n              - key: environment_sh\n                path: '\"'\"'environment.sh'\"'\"'\n\n        - name: awx-secret-key\n          secret:\n            secretName: \"awx-secrets\"\n            items:\n              - key: secret_key\n                path: SECRET_KEY\n\n        - name: rabbitmq-config\n          configMap:\n            name: rabbitmq-config\n            items:\n            - key: rabbitmq.conf\n              path: rabbitmq.conf\n            - key: enabled_plugins\n              path: enabled_plugins\n            - key: rabbitmq_definitions.json\n              path: rabbitmq_definitions.json\n        - name: rabbitmq-healthchecks\n          configMap:\n            name: awx-healthchecks\n            items:\n            - key: rabbit_health_node.py\n              path: rabbit_health_node.py\n            defaultMode: 0755\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: awx-healthchecks\n  namespace: awx\ndata:\n  rabbit_health_node.py: |\n    #!/usr/bin/env python\n    try:\n      from http.client import HTTPConnection\n    except ImportError:\n      from httplib import HTTPConnection\n    import sys\n    import os\n    import base64\n    authsecret = base64.b64encode(os.getenv('\"'\"'RABBITMQ_USER'\"'\"') + '\"'\"':'\"'\"' + os.getenv('\"'\"'RABBITMQ_PASSWORD'\"'\"'))\n    conn=HTTPConnection('\"'\"'localhost:15672'\"'\"')\n    conn.request('\"'\"'GET'\"'\"', '\"'\"'/api/healthchecks/node'\"'\"', headers={'\"'\"'Authorization'\"'\"': '\"'\"'Basic %s'\"'\"' % authsecret})\n    r1 = conn.getresponse()\n    if r1.status != 200:\n      sys.stderr.write('\"'\"'Received http error %i\\n'\"'\"' % (r1.status))\n      sys.exit(1)\n    body = r1.read()\n    if body != '\"'\"'{\"status\":\"ok\"}'\"'\"':\n      sys.stderr.write('\"'\"'Received body: %s'\"'\"' % body)\n      sys.exit(2)\n    sys.exit(0)\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: awx-web-svc\n  namespace: awx\n  labels:\n    name: awx-web-svc\nspec:\n  type: \"NodePort\"\n  ports:\n    - name: http\n      port: 80\n      targetPort: 8052\n  selector:\n    name: awx-web-deploy\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: awx-rmq-mgmt\n  namespace: awx\n  labels:\n    name: awx-rmq-mgmt\nspec:\n  type: ClusterIP\n  ports:\n    - name: rmqmgmt\n      port: 15672\n      targetPort: 15672\n  selector:\n    name: awx-web-deploy\n---\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  name: awx-web-svc\n  namespace: awx\nspec:\n  backend:\n    serviceName: awx-web-svc\n    servicePort: 80\n' | kubectl apply -f -", "delta": "0:00:01.443431", "end": "2019-11-03 16:58:24.757486", "item": "---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: awx\n  namespace: awx\n\n---\nkind: Service\napiVersion: v1\nmetadata:\n  namespace: awx\n  name: rabbitmq\n  labels:\n    app: awx\n    type: LoadBalancer\nspec:\n  type: NodePort\n  ports:\n   - name: http\n     protocol: TCP\n     port: 15672\n     targetPort: 15672\n   - name: amqp\n     protocol: TCP\n     port: 5672\n     targetPort: 5672\n  selector:\n    app: awx\n\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: rabbitmq-config\n  namespace: awx\ndata:\n  enabled_plugins: |\n      [rabbitmq_management,rabbitmq_peer_discovery_k8s].\n  rabbitmq_definitions.json: |\n      {\n        \"users\":[{\"name\": \"awx\", \"password\": \"awxpass\", \"tags\": \"administrator\"}],\n        \"permissions\":[\n          {\"user\":\"awx\",\"vhost\":\"awx\",\"configure\":\".*\",\"write\":\".*\",\"read\":\".*\"}\n        ],\n        \"vhosts\":[{\"name\":\"awx\"}],\n        \"policies\":[\n            {\"vhost\":\"awx\",\"name\":\"ha-all\",\"pattern\":\".*\",\"definition\":{\"ha-mode\":\"all\",\"ha-sync-mode\":\"automatic\"}}\n        ]\n      }\n  rabbitmq.conf: |\n      ## Clustering\n      management.load_definitions = /etc/rabbitmq/rabbitmq_definitions.json\n      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s\n      cluster_formation.k8s.host = kubernetes.default.svc\n      cluster_formation.k8s.address_type = ip\n      cluster_formation.node_cleanup.interval = 10\n      cluster_formation.node_cleanup.only_log_warning = false\n      cluster_partition_handling = autoheal\n      ## queue master locator\n      queue_master_locator=min-masters\n      ## enable guest user\n      loopback_users.guest = false\n\n---\nkind: Role\napiVersion: rbac.authorization.k8s.io/v1beta1\nmetadata:\n  name: endpoint-reader\n  namespace: awx\nrules:\n- apiGroups: [\"\"]\n  resources: [\"endpoints\"]\n  verbs: [\"get\"]\n---\nkind: RoleBinding\napiVersion: rbac.authorization.k8s.io/v1beta1\nmetadata:\n  name: endpoint-reader\n  namespace: awx\nsubjects:\n- kind: ServiceAccount\n  name: awx\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: Role\n  name: endpoint-reader\n\n\n---\napiVersion: apps/v1beta1\nkind: StatefulSet\nmetadata:\n  name: awx\n  namespace: awx\nspec:\n  serviceName: awx\n  replicas: 1\n  template:\n    metadata:\n      labels:\n        name: awx-web-deploy\n        service: django\n        app: awx\n    spec:\n      serviceAccountName: awx\n      terminationGracePeriodSeconds: 10\n      containers:\n        - name: awx-web\n          image: \"ansible/awx_web:8.0.0\"\n          imagePullPolicy: Always\n          ports:\n            - containerPort: 8052\n          volumeMounts:\n            - name: awx-application-config\n              mountPath: \"/etc/tower/settings.py\"\n              subPath: settings.py\n              readOnly: true\n\n            - name: \"awx-application-credentials\"\n              mountPath: \"/etc/tower/conf.d/\"\n              readOnly: true\n\n            - name: awx-secret-key\n              mountPath: \"/etc/tower/SECRET_KEY\"\n              subPath: SECRET_KEY\n              readOnly: true\n\n          resources:\n            requests:\n              memory: \"1Gi\"\n              cpu: \"500m\"\n        - name: awx-celery\n          securityContext:\n            privileged: true\n          image: \"ansible/awx_task:8.0.0\"\n          command:\n            - /usr/bin/launch_awx_task.sh\n          imagePullPolicy: Always\n          volumeMounts:\n            - name: awx-application-config\n              mountPath: \"/etc/tower/settings.py\"\n              subPath: settings.py\n              readOnly: true\n\n            - name: \"awx-application-credentials\"\n              mountPath: \"/etc/tower/conf.d/\"\n              readOnly: true\n\n            - name: awx-secret-key\n              mountPath: \"/etc/tower/SECRET_KEY\"\n              subPath: SECRET_KEY\n              readOnly: true\n          env:\n            - name: AWX_SKIP_MIGRATIONS\n              value: \"1\"\n          resources:\n            requests:\n              memory: \"2Gi\"\n              cpu: \"1500m\"\n        - name: awx-rabbit\n          image: \"ansible/awx_rabbitmq:3.7.4\"\n          imagePullPolicy: Always\n          ports:\n            - name: http\n              protocol: TCP\n              containerPort: 15672\n            - name: amqp\n              protocol: TCP\n              containerPort: 5672\n          livenessProbe:\n            exec:\n              command:\n              - /usr/local/bin/healthchecks/rabbit_health_node.py\n            initialDelaySeconds: 30\n            timeoutSeconds: 10\n          readinessProbe:\n            exec:\n              command:\n              - /usr/local/bin/healthchecks/rabbit_health_node.py\n            initialDelaySeconds: 10\n            timeoutSeconds: 10\n          env:\n            - name: MY_POD_IP\n              valueFrom:\n                fieldRef:\n                  fieldPath: status.podIP\n            - name: RABBITMQ_USE_LONGNAME\n              value: \"true\"\n            - name: RABBITMQ_NODENAME\n              value: \"rabbit@$(MY_POD_IP)\"\n            - name: RABBITMQ_ERLANG_COOKIE\n              valueFrom:\n                secretKeyRef:\n                  name: \"awx-secrets\"\n                  key: rabbitmq_erlang_cookie\n            - name: K8S_SERVICE_NAME\n              value: \"rabbitmq\"\n            - name: RABBITMQ_USER\n              value: awx\n            - name: RABBITMQ_PASSWORD\n              valueFrom:\n                secretKeyRef:\n                  name: \"awx-secrets\"\n                  key: rabbitmq_password\n          volumeMounts:\n            - name: rabbitmq-config\n              mountPath: /etc/rabbitmq\n            - name: rabbitmq-healthchecks\n              mountPath: /usr/local/bin/healthchecks\n          resources:\n            requests:\n              memory: \"2Gi\"\n              cpu: \"500m\"\n        - name: awx-memcached\n          image: \"memcached:latest\"\n          imagePullPolicy: Always\n          resources:\n            requests:\n              memory: \"1Gi\"\n              cpu: \"500m\"\n      volumes:\n        - name: awx-application-config\n          configMap:\n            name: awx-config\n            items:\n              - key: awx_settings\n                path: settings.py\n\n        - name: \"awx-application-credentials\"\n          secret:\n            secretName: \"awx-secrets\"\n            items:\n              - key: credentials_py\n                path: 'credentials.py'\n              - key: environment_sh\n                path: 'environment.sh'\n\n        - name: awx-secret-key\n          secret:\n            secretName: \"awx-secrets\"\n            items:\n              - key: secret_key\n                path: SECRET_KEY\n\n        - name: rabbitmq-config\n          configMap:\n            name: rabbitmq-config\n            items:\n            - key: rabbitmq.conf\n              path: rabbitmq.conf\n            - key: enabled_plugins\n              path: enabled_plugins\n            - key: rabbitmq_definitions.json\n              path: rabbitmq_definitions.json\n        - name: rabbitmq-healthchecks\n          configMap:\n            name: awx-healthchecks\n            items:\n            - key: rabbit_health_node.py\n              path: rabbit_health_node.py\n            defaultMode: 0755\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: awx-healthchecks\n  namespace: awx\ndata:\n  rabbit_health_node.py: |\n    #!/usr/bin/env python\n    try:\n      from http.client import HTTPConnection\n    except ImportError:\n      from httplib import HTTPConnection\n    import sys\n    import os\n    import base64\n    authsecret = base64.b64encode(os.getenv('RABBITMQ_USER') + ':' + os.getenv('RABBITMQ_PASSWORD'))\n    conn=HTTPConnection('localhost:15672')\n    conn.request('GET', '/api/healthchecks/node', headers={'Authorization': 'Basic %s' % authsecret})\n    r1 = conn.getresponse()\n    if r1.status != 200:\n      sys.stderr.write('Received http error %i\\n' % (r1.status))\n      sys.exit(1)\n    body = r1.read()\n    if body != '{\"status\":\"ok\"}':\n      sys.stderr.write('Received body: %s' % body)\n      sys.exit(2)\n    sys.exit(0)\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: awx-web-svc\n  namespace: awx\n  labels:\n    name: awx-web-svc\nspec:\n  type: \"NodePort\"\n  ports:\n    - name: http\n      port: 80\n      targetPort: 8052\n  selector:\n    name: awx-web-deploy\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: awx-rmq-mgmt\n  namespace: awx\n  labels:\n    name: awx-rmq-mgmt\nspec:\n  type: ClusterIP\n  ports:\n    - name: rmqmgmt\n      port: 15672\n      targetPort: 15672\n  selector:\n    name: awx-web-deploy\n---\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n  name: awx-web-svc\n  namespace: awx\nspec:\n  backend:\n    serviceName: awx-web-svc\n    servicePort: 80\n", "msg": "non-zero return code", "rc": 1, "start": "2019-11-03 16:58:23.314055", "stderr": "error: error parsing STDIN: error converting YAML to JSON: yaml: line 25: could not find expected ':'", "stderr_lines": ["error: error parsing STDIN: error converting YAML to JSON: yaml: line 25: could not find expected ':'"], "stdout": "serviceaccount/awx created\nservice/rabbitmq created\nconfigmap/rabbitmq-config created\nrole.rbac.authorization.k8s.io/endpoint-reader created\nrolebinding.rbac.authorization.k8s.io/endpoint-reader created\nstatefulset.apps/awx created", "stdout_lines": ["serviceaccount/awx created", "service/rabbitmq created", "configmap/rabbitmq-config created", "role.rbac.authorization.k8s.io/endpoint-reader created", "rolebinding.rbac.authorization.k8s.io/endpoint-reader created", "statefulset.apps/awx created"]}
changed: [localhost] => (item=---
apiVersion: v1
kind: Secret
metadata:
  namespace: awx
  name: "awx-secrets"
type: Opaque
data:
  secret_key: "<snip>"
  admin_password: "<snip>"
  pg_password: "<snip>"
  rabbitmq_password: "<snip>"
  rabbitmq_erlang_cookie: "<snip>"
  credentials_py: "<snip>"
  environment_sh: "<snip>"
)
        to retry, use: --limit @/mnt/c/Users/<snip>/Code/awx8/installer/install.retry

PLAY RECAP *********************************************************************************************************************************************************************************************************
localhost                  : ok=18   changed=5    unreachable=0    failed=1
ADDITIONAL INFORMATION

Using the exact same steps above, the install is successful on the 7.0.0 tag.

shanemcd commented 4 years ago

I just pushed up AWX 9.0.1. Can you try now? Please ensure you download the correct version from the releases page (https://github.com/ansible/awx/releases) and repopulate the inventory file.

Quantas commented 4 years ago

I’ll give it a shot this afternoon and report back, thank you for the fast responses on the tickets I filed this weekend!

Quantas commented 4 years ago

@shanemcd 9.0.1 works perfect on a fresh install! Thank you for the extremely quick turn around!