AWX is upgrading

[nicolaibaralmueller]

Hi Jeff,

AWX is stuck at "AWX is Upgrading" when using your guide.

Not sure how to debug.

Thanks

[geerlingguy]

@nicolaibaralmueller - Note that the 'AWX is Upgrading' thing can take a long time—sometimes I've clocked it at 5 or 10 minutes, depending on the speed of the server(s) running the cluster.

But you can monitor the logs of the pod awx-task to see what's happening, via kubectl logs -f -n [awx/tower namespace] [pod id] and then watch what is happening. If it shows things like "schema not present" or something like that, it could still be building the database tables.

Also check on all the pods and make sure they're running, and none are resource constrained. You can do kubectl get pods --all-namespaces and see if any are not in 'Running' state. If any are not Running, do a kubectl describe pod -n [namespace] [pod id] and see why it's not Running.

[nicolaibaralmueller]

@nicolaibaralmueller - Note that the 'AWX is Upgrading' thing can take a long time—sometimes I've clocked it at 5 or 10 minutes, depending on the speed of the server(s) running the cluster.

But you can monitor the logs of the pod awx-task to see what's happening, via kubectl logs -f -n [awx/tower namespace] [pod id] and then watch what is happening. If it shows things like "schema not present" or something like that, it could still be building the database tables.

Also check on all the pods and make sure they're running, and none are resource constrained. You can do kubectl get pods --all-namespaces and see if any are not in 'Running' state. If any are not Running, do a kubectl describe pod -n [namespace] [pod id] and see why it's not Running.

Hi Jeff,

All pods are running. Checked in this morning and AWX is still in an upgrading state.

Log output shows:

[pid: 126|app: 0|req: 30/140] 10.1.90.1 () {58 vars in 952 bytes} [Fri Mar 13 07:57:44 2020] GET /api/ => generated 0 bytes in 41 msecs (HTTP/1.1 302) 3 headers in 119 bytes (1 switches on core 0)
10.1.90.1 - - [13/Mar/2020:07:57:44 +0000] "GET /api/ HTTP/1.1" 302 5 "http://minikube/migrations_notran/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) snap Chromium/80.0.3987.132 Chrome/80.0.3987.132 Safari/537.36" "10.199.8.152"
2020-03-13 07:57:44,435 ERROR    awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "conf_setting" does not exist
LINE 1: ...f_setting"."value", "conf_setting"."user_id" FROM "conf_sett...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 87, in _ctit_db_wrapper
    yield
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 415, in __getattr__
    value = self._get_local(name)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 358, in _get_local
    setting = Setting.objects.filter(key=name, user__isnull=True).order_by('pk').first()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 653, in first
    for obj in (self if self.ordered else self.order_by('pk'))[:1]:
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "conf_setting" does not exist
LINE 1: ...f_setting"."value", "conf_setting"."user_id" FROM "conf_sett...

Describe:

microk8s.kubectl describe pods -n awx
Name:         awx-memcached-587b55d5fd-8r8fl
Namespace:    awx
Priority:     0
Node:         minikube/192.168.0.6
Start Time:   Thu, 12 Mar 2020 11:03:41 +0000
Labels:       app=tower-memcached
              pod-template-hash=587b55d5fd
Annotations:  <none>
Status:       Running
IP:           10.1.90.129
IPs:
  IP:           10.1.90.129
Controlled By:  ReplicaSet/awx-memcached-587b55d5fd
Containers:
  memcached:
    Container ID:   containerd://e317f53b974a2d1ff43946e601b694dd75dd92032f1d6e9cbb00a0e0f5922cee
    Image:          memcached:alpine
    Image ID:       docker.io/library/memcached@sha256:94a32becfb362d4f5dc648da4ee110ca3d057e56b784bfa2c95c04ee6f13138b
    Port:           11211/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 13 Mar 2020 07:52:58 +0000
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Thu, 12 Mar 2020 13:28:16 +0000
      Finished:     Fri, 13 Mar 2020 07:51:43 +0000
    Ready:          True
    Restart Count:  3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qss58 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-qss58:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qss58
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  SandboxChanged  6m42s  kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          6m27s  kubelet, minikube  Container image "memcached:alpine" already present on machine
  Normal  Created         6m21s  kubelet, minikube  Created container memcached
  Normal  Started         6m18s  kubelet, minikube  Started container memcached

Name:         awx-postgres-0
Namespace:    awx
Priority:     0
Node:         minikube/192.168.0.6
Start Time:   Thu, 12 Mar 2020 11:03:44 +0000
Labels:       app=tower-postgres
              controller-revision-hash=awx-postgres-5599b677
              statefulset.kubernetes.io/pod-name=awx-postgres-0
Annotations:  <none>
Status:       Running
IP:           10.1.90.127
IPs:
  IP:           10.1.90.127
Controlled By:  StatefulSet/awx-postgres
Containers:
  postgres:
    Container ID:   containerd://a606a5d1d2db0c924196684e350e857fd2e8216e953ce9313542cb913c0aaa89
    Image:          postgres:10
    Image ID:       docker.io/library/postgres@sha256:73d3ac7b17b8cd2122d27026ec3552080e8aaea95fef0b6e671fa795ac547f94
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 13 Mar 2020 07:52:57 +0000
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Thu, 12 Mar 2020 13:28:17 +0000
      Finished:     Fri, 13 Mar 2020 07:51:43 +0000
    Ready:          True
    Restart Count:  3
    Environment:
      POSTGRES_DB:        awx
      POSTGRES_USER:      awx
      POSTGRES_PASSWORD:  <set to the key 'password' in secret 'awx-postgres-pass'>  Optional: false
    Mounts:
      /var/lib/postgresql/data from postgres (rw,path="data")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qss58 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  postgres:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-awx-postgres-0
    ReadOnly:   false
  default-token-qss58:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qss58
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  SandboxChanged  6m42s  kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          6m39s  kubelet, minikube  Container image "postgres:10" already present on machine
  Normal  Created         6m25s  kubelet, minikube  Created container postgres
  Normal  Started         6m18s  kubelet, minikube  Started container postgres

Name:         awx-rabbitmq-7f8f6ff647-wpf7b
Namespace:    awx
Priority:     0
Node:         minikube/192.168.0.6
Start Time:   Thu, 12 Mar 2020 11:03:44 +0000
Labels:       app=tower-rabbitmq
              pod-template-hash=7f8f6ff647
Annotations:  <none>
Status:       Running
IP:           10.1.90.125
IPs:
  IP:           10.1.90.125
Controlled By:  ReplicaSet/awx-rabbitmq-7f8f6ff647
Containers:
  rabbitmq:
    Container ID:   containerd://e0de96cff729fb049750246c01fc02fc951e2390831e871634e912cdb5227650
    Image:          rabbitmq:3
    Image ID:       docker.io/library/rabbitmq@sha256:788965e47cb8ba64e46c00f493358be888057db9a14eca93724a4d71bbffd27f
    Ports:          15672/TCP, 5672/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Fri, 13 Mar 2020 07:52:42 +0000
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Thu, 12 Mar 2020 13:28:16 +0000
      Finished:     Fri, 13 Mar 2020 07:51:41 +0000
    Ready:          True
    Restart Count:  3
    Environment:
      RABBITMQ_DEFAULT_VHOST:  awx
      RABBITMQ_NODE_PORT:      5672
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qss58 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-qss58:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qss58
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  SandboxChanged  6m42s  kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          6m40s  kubelet, minikube  Container image "rabbitmq:3" already present on machine
  Normal  Created         6m39s  kubelet, minikube  Created container rabbitmq
  Normal  Started         6m33s  kubelet, minikube  Started container rabbitmq

Name:         awx-tower-web-5b7bc9f756-llr9h
Namespace:    awx
Priority:     0
Node:         minikube/192.168.0.6
Start Time:   Thu, 12 Mar 2020 11:03:47 +0000
Labels:       app=tower
              pod-template-hash=5b7bc9f756
Annotations:  <none>
Status:       Running
IP:           10.1.90.128
IPs:
  IP:           10.1.90.128
Controlled By:  ReplicaSet/awx-tower-web-5b7bc9f756
Containers:
  tower:
    Container ID:   containerd://45f3b141fae32d01a2b3e19fc81f1ee9b4bf1b3fa8f7b4434336400df7e8f838
    Image:          ansible/awx_web:9.2.0
    Image ID:       docker.io/ansible/awx_web@sha256:57232e6820eb1bfad3c6910bd993e1a8c0d644c0c0fcfb75a25a0b51b3d6fdec
    Port:           8052/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 13 Mar 2020 07:53:06 +0000
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Thu, 12 Mar 2020 13:28:15 +0000
      Finished:     Fri, 13 Mar 2020 07:51:43 +0000
    Ready:          True
    Restart Count:  3
    Requests:
      cpu:        1
      memory:     2Gi
    Environment:  <none>
    Mounts:
      /etc/nginx/nginx.conf from nginx-conf (ro,path="nginx.conf")
      /etc/tower/SECRET_KEY from secret-key (ro,path="SECRET_KEY")
      /etc/tower/conf.d/environment.sh from environment (ro,path="environment.sh")
      /etc/tower/settings.py from settings (ro,path="settings.py")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qss58 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  secret-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-tower-secret
    Optional:    false
  environment:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-tower-configmap
    Optional:  false
  settings:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-tower-configmap
    Optional:  false
  nginx-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-tower-configmap
    Optional:  false
  default-token-qss58:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qss58
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  SandboxChanged  6m42s  kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          6m30s  kubelet, minikube  Container image "ansible/awx_web:9.2.0" already present on machine
  Normal  Created         6m15s  kubelet, minikube  Created container tower
  Normal  Started         6m9s   kubelet, minikube  Started container tower

[tylerauerbeck]

@nicolaibaralmueller Can you provide the logs from the ansible container of the operator (i.e. kubectl logs -c ansible)? I'm seeing something similar and I noticed this in my logs:

failed: [localhost] (item=tower_web.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": "tower_web.yaml.j2", "msg": "Failed to create object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"ingresses.extensions \\\\\"ta-tower-ingress\\\\\" is invalid: spec.rules[0]: Forbidden: you do not have permission to set host fields in ingress rules\",\"reason\":\"Invalid\",\"details\":{\"name\":\"ta-tower-ingress\",\"group\":\"extensions\",\"kind\":\"ingresses\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: you do not have permission to set host fields in ingress rules\",\"field\":\"spec.rules[0]\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}

Because of this the play fails and doesn't get past processing the templates

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0 

And for that reason, I don't think the database setup is ever actually run (which is why it stays in updating/upgrading. Can't say that this will be your exact scenario, but something to look into.

cc/ @geerlingguy Interested on whether this is by design or if this is something that needs accounted for.

[geerlingguy]

Forbidden: you do not have permission to set host fields in ingress rules

@tylerauerbeck - Ah, that could definitely be it; Tower setup won't complete if it bails during the deployment. In which case if you do a describe on the Tower CR, it should say it's still trying to set up, or failing, something like that.

Are you installing on Kubernetes or OpenShift?

If it's OpenShift, it looks like this is some new restriction that was added in 3.x that requires a 'Pro' plan instead of a 'Starter' plan (https://stackoverflow.com/a/47105700/100134). Note that I'm not sure the implications for 4.x. I've never seen this error in any Kubernetes cluster though.

If this is going to be a blocker in OpenShift, maybe we can make the entire ingress configuration optional? E.g. if the host var is empty, don't add ingress.

[tylerauerbeck]

@geerlingguy I'm running this on OpenShift 4.X cluster, but this is a self hosted cluster.. so the hosted restrictions don't apply here. I'm going to dig in here to see why I'm not able to do this (whether this is a permissions piece or if this is actually a restriction on the object itself) and I'll report back. I think the flip side of this is if this is specifically an OpenShift thing.. do you want to add the ability to just work with Route objects directly?

@nicolaibaralmueller I don't want to hijack this thread, so please feel free to provide your logs and we can dig in on your issue as well.

[tylerauerbeck]

@geerlingguy Little bit of an update here. So I've found that if I try to create that exact Ingress directly, I don't have an issue. So this leads me to believe this is just an RBAC issue of some sort with the tower-operator serviceaccount. That being said I'm trying to focus on getting something up and running with OpenShift versus trying to make it perfect out of the gate. So I think for now I'm going to follow your initial idea of skipping the Ingress configuration if the host var is empty. Once I've done that, everything proceeds as expected.

[nicolaibaralmueller]

microk8s.kubectl logs tower-operator-7cbd9895cd-nxt7f -c ansible
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ansible-operator/runner/tower.ansible.com/v1alpha1/Tower/awx/awx/artifacts/6129484611666145821//stdout
ansible-playbook 2.9.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: main.yml *************************************************************
1 plays in /opt/ansible/main.yml

PLAY [localhost] ***************************************************************
META: ran handlers

TASK [tower : Ensure configured Tower resources exist in the cluster.] *********
task path: /opt/ansible/roles/tower/tasks/main.yml:2
changed: [localhost] => (item=tower_memcached.yaml.j2) => {"ansible_loop_var": "item", "changed": true, "item": "tower_memcached.yaml.j2", "result": {"results": [{"changed": true, "method": "create", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"creationTimestamp": "2020-03-16T10:18:24Z", "generation": 1, "labels": {"app": "tower-memcached"}, "name": "awx-memcached", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958066", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-memcached", "uid": "354b9c58-3453-4a5b-ba49-2802341eb4fe"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-memcached"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-memcached"}}, "spec": {"containers": [{"image": "memcached:alpine", "imagePullPolicy": "IfNotPresent", "name": "memcached", "ports": [{"containerPort": 11211, "protocol": "TCP"}], "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File"}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30}}}, "status": {}}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:24Z", "labels": {"app": "tower-memcached"}, "name": "awx-memcached", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958079", "selfLink": "/api/v1/namespaces/awx/services/awx-memcached", "uid": "10eafa1e-6c1f-4441-9f75-fa5a9d42540d"}, "spec": {"clusterIP": "None", "ports": [{"port": 11211, "protocol": "TCP", "targetPort": 11211}], "selector": {"app": "tower-memcached"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}]}}
changed: [localhost] => (item=tower_postgres.yaml.j2) => {"ansible_loop_var": "item", "changed": true, "item": "tower_postgres.yaml.j2", "result": {"results": [{"changed": true, "method": "create", "result": {"apiVersion": "v1", "data": {"password": "YXd4cGFzcw=="}, "kind": "Secret", "metadata": {"creationTimestamp": "2020-03-16T10:18:25Z", "name": "awx-postgres-pass", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958087", "selfLink": "/api/v1/namespaces/awx/secrets/awx-postgres-pass", "uid": "59a0c17a-4829-40ae-972f-f66cc86ae4f8"}, "type": "Opaque"}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "apps/v1", "kind": "StatefulSet", "metadata": {"creationTimestamp": "2020-03-16T10:18:26Z", "generation": 1, "labels": {"app": "tower-postgres"}, "name": "awx-postgres", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958093", "selfLink": "/apis/apps/v1/namespaces/awx/statefulsets/awx-postgres", "uid": "3d64a660-2bfb-4b52-9473-db66446509c3"}, "spec": {"podManagementPolicy": "OrderedReady", "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-postgres"}}, "serviceName": "awx", "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-postgres"}}, "spec": {"containers": [{"env": [{"name": "POSTGRES_DB", "value": "awx"}, {"name": "POSTGRES_USER", "value": "awx"}, {"name": "POSTGRES_PASSWORD", "valueFrom": {"secretKeyRef": {"key": "password", "name": "awx-postgres-pass"}}}], "image": "postgres:10", "imagePullPolicy": "IfNotPresent", "name": "postgres", "ports": [{"containerPort": 3306, "name": "postgres", "protocol": "TCP"}], "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/var/lib/postgresql/data", "name": "postgres", "subPath": "data"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30}}, "updateStrategy": {"type": "RollingUpdate"}, "volumeClaimTemplates": [{"apiVersion": "v1", "kind": "PersistentVolumeClaim", "metadata": {"creationTimestamp": null, "name": "postgres"}, "spec": {"accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "8Gi"}}, "volumeMode": "Filesystem"}, "status": {"phase": "Pending"}}]}, "status": {"replicas": 0}}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:26Z", "labels": {"app": "tower-postgres"}, "name": "awx-postgres", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958096", "selfLink": "/api/v1/namespaces/awx/services/awx-postgres", "uid": "4ceb15a7-fee6-4ebb-aa36-82c832661c67"}, "spec": {"clusterIP": "None", "ports": [{"port": 5432, "protocol": "TCP", "targetPort": 5432}], "selector": {"app": "tower-postgres"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}]}}
changed: [localhost] => (item=tower_rabbitmq.yaml.j2) => {"ansible_loop_var": "item", "changed": true, "item": "tower_rabbitmq.yaml.j2", "result": {"results": [{"changed": true, "method": "create", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"creationTimestamp": "2020-03-16T10:18:27Z", "generation": 1, "labels": {"app": "tower-rabbitmq"}, "name": "awx-rabbitmq", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958120", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-rabbitmq", "uid": "1c3e3bc9-f8e3-4ddb-ab69-02b5fe842dce"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-rabbitmq"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-rabbitmq"}}, "spec": {"containers": [{"env": [{"name": "RABBITMQ_DEFAULT_VHOST", "value": "awx"}, {"name": "RABBITMQ_NODE_PORT", "value": "5672"}], "image": "rabbitmq:3", "imagePullPolicy": "IfNotPresent", "name": "rabbitmq", "ports": [{"containerPort": 15672, "protocol": "TCP"}, {"containerPort": 5672, "protocol": "TCP"}], "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File"}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30}}}, "status": {}}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:27Z", "labels": {"app": "tower-rabbitmq"}, "name": "awx-rabbitmq", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958125", "selfLink": "/api/v1/namespaces/awx/services/awx-rabbitmq", "uid": "b811d4df-9277-485c-b946-588a8e60a8d5"}, "spec": {"clusterIP": "None", "ports": [{"port": 5672, "protocol": "TCP", "targetPort": 5672}], "selector": {"app": "tower-rabbitmq"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}]}}
 \"default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/\";\n\n        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)\n        add_header X-Frame-Options \"DENY\";\n\n        location /nginx_status {\n            stub_status on;\n            access_log off;\n            allow 127.0.0.1;\n            deny all;\n        }\n\n        location /static/ {\n            alias /var/lib/awx/public/static/;\n        }\n\n        location /favicon.ico {\n            alias /var/lib/awx/public/static/favicon.ico;\n        }\n\n        location /websocket {\n            # Pass request to the upstream alias\n            proxy_pass http://daphne;\n            # Require http version 1.1 to allow for upgrade requests\n            proxy_http_version 1.1;\n            # We want proxy_buffering off for proxying to websockets.\n            proxy_buffering off;\n            # http://en.wikipedia.org/wiki/X-Forwarded-For\n            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n            # enable this if you use HTTPS:\n            proxy_set_header X-Forwarded-Proto https;\n            # pass the Host: header from the client for the sake of redirects\n            proxy_set_header Host $http_host;\n            # We've set the Host header, so we don't need Nginx to muddle\n            # about with redirects\n            proxy_redirect off;\n            # Depending on the request value, set the Upgrade and\n            # connection headers\n            proxy_set_header Upgrade $http_upgrade;\n            proxy_set_header Connection $connection_upgrade;\n        }\n\n        location / {\n            # Add trailing / if missing\n            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;\n            uwsgi_read_timeout 120s;\n            uwsgi_pass uwsgi;\n            include /etc/nginx/uwsgi_params;                proxy_set_header X-Forwarded-Port 443;\n        }\n    }\n}\n", "settings": "import os\nimport socket\n\ndef get_secret():\n    if os.path.exists(\"/etc/tower/SECRET_KEY\"):\n        return open('/etc/tower/SECRET_KEY', 'rb').read().strip()\n\nADMINS = ()\nSTATIC_ROOT = '/var/lib/awx/public/static'\nPROJECTS_ROOT = '/var/lib/awx/projects'\nJOBOUTPUT_ROOT = '/var/lib/awx/job_status'\n\nSECRET_KEY = get_secret()\n\nALLOWED_HOSTS = ['*']\n\nINTERNAL_API_URL = 'http://127.0.0.1:8052'\n\n# Container environments don't like chroots\nAWX_PROOT_ENABLED = False\n\n# Automatically deprovision pods that go offline\nAWX_AUTO_DEPROVISION_INSTANCES = True\n\nCLUSTER_HOST_ID = socket.gethostname()\nSYSTEM_UUID = '00000000-0000-0000-0000-000000000000'\n\nCSRF_COOKIE_SECURE = False\nSESSION_COOKIE_SECURE = False\n\nSERVER_EMAIL = 'root@localhost'\nDEFAULT_FROM_EMAIL = 'webmaster@localhost'\nEMAIL_SUBJECT_PREFIX = '[AWX] '\n\nEMAIL_HOST = 'localhost'\nEMAIL_PORT = 25\nEMAIL_HOST_USER = ''\nEMAIL_HOST_PASSWORD = ''\nEMAIL_USE_TLS = False\n\nLOGGING['handlers']['console'] = {\n    '()': 'logging.StreamHandler',\n    'level': 'DEBUG',\n    'formatter': 'simple',\n}\n\nLOGGING['loggers']['django.request']['handlers'] = ['console']\nLOGGING['loggers']['rest_framework.request']['handlers'] = ['console']\nLOGGING['loggers']['awx']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.commands.run_callback_receiver']['handlers'] = ['console']\nLOGGING['loggers']['awx.main.tasks']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.scheduler']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['django_auth_ldap']['handlers'] = ['console']\nLOGGING['loggers']['social']['handlers'] = ['console']\nLOGGING['loggers']['system_tracking_migrations']['handlers'] = ['console']\nLOGGING['loggers']['rbac_migrations']['handlers'] = ['console']\nLOGGING['loggers']['awx.isolated.manager.playbooks']['handlers'] = ['console']\nLOGGING['handlers']['callback_receiver'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['task_system'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['tower_warnings'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['rbac_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['system_tracking_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['management_playbooks'] = {'class': 'logging.NullHandler'}\n\nDATABASES = {\n    'default': {\n        'ATOMIC_REQUESTS': True,\n        'ENGINE': 'awx.main.db.profiled_pg',\n        'NAME': 'awx',\n        'USER': 'awx',\n        'PASSWORD': 'awxpass',\n        'HOST': 'awx-postgres.awx.svc.cluster.local',\n        'PORT': '5432',\n    }\n}\n\nif os.getenv(\"DATABASE_SSLMODE\", False):\n    DATABASES['default']['OPTIONS'] = {'sslmode': os.getenv(\"DATABASE_SSLMODE\")}\n\nCACHES = {\n    'default': {\n        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',\n        'LOCATION': '{}:{}'.format(\"awx-memcached.awx.svc.cluster.local\", \"11211\")\n    },\n    'ephemeral': {\n        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',\n    },\n}\n\nBROKER_URL = 'amqp://{}:{}@{}:{}/{}'.format(\n    'guest',\n    'guest',\n    'awx-rabbitmq.awx.svc.cluster.local',\n    '5672',\n    'awx')\n\nCHANNEL_LAYERS = {\n    'default': {'BACKEND': 'asgi_amqp.AMQPChannelLayer',\n                'ROUTING': 'awx.main.routing.channel_routing',\n                'CONFIG': {'url': BROKER_URL}}\n}\n\nUSE_X_FORWARDED_PORT = True\n"}, "kind": "ConfigMap", "metadata": {"creationTimestamp": "2020-03-16T10:18:28Z", "labels": {"app": "tower"}, "name": "awx-tower-configmap", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958144", "selfLink": "/api/v1/namespaces/awx/configmaps/awx-tower-configmap", "uid": "399ccb43-8c32-4changed: [localhost] => (item=tower_config.yaml.j2) => {"ansible_loop_var": "item", "changed": true, "item": "tower_config.yaml.j2", "method": "create", "result": {"apiVersion": "v1", "data": {"environment": "DATABASE_USER=awx\nDATABASE_NAME=awx\nDATABASE_HOST='awx-postgres.awx.svc.cluster.local'\nDATABASE_PORT='5432'\nDATABASE_PASSWORD=awxpass\nMEMCACHED_HOST='awx-memcached.awx.svc.cluster.local'\nMEMCACHED_PORT='11211'\nRABBITMQ_HOST='awx-rabbitmq.awx.svc.cluster.local'\nRABBITMQ_PORT='5672'\nAWX_SKIP_MIGRATIONS=true\n", "nginx_conf": "worker_processes  1;\npid        /tmp/nginx.pid;\n\nevents {\n    worker_connections  1024;\n}\n\nhttp {\n    include       /etc/nginx/mime.types;\n    default_type  application/octet-stream;\n    server_tokens off;\n\n    log_format  main  '$remote_addr - $remote_user [$time_local] \"$request\" '\n                      '$status $body_bytes_sent \"$http_referer\" '\n                      '\"$http_user_agent\" \"$http_x_forwarded_for\"';\n\n    access_log /dev/stdout main;\n\n    map $http_upgrade $connection_upgrade {\n        default upgrade;\n        ''      close;\n    }\n\n    sendfile        on;\n    #tcp_nopush     on;\n    #gzip  on;\n\n    upstream uwsgi {\n        server 127.0.0.1:8050;\n    }\n\n    upstream daphne {\n        server 127.0.0.1:8051;\n    }\n\n    server {\n        listen 8052 default_server;\n\n        # If you have a domain name, this is where to add it\n        server_name _;\n        keepalive_timeout 65;\n\n        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)\n        add_header Strict-Transport-Security max-age=15768000;\n        add_header Content-Security-Policy \"default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/\";\n        add_header X-Content-Security-Policy \"default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/\";\n\n        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)\n        add_header X-Frame-Options \"DENY\";\n\n        location /nginx_status {\n            stub_status on;\n            access_log off;\n            allow 127.0.0.1;\n            deny all;\n        }\n\n        location /static/ {\n            alias /var/lib/awx/public/static/;\n        }\n\n        location /favicon.ico {\n            alias /var/lib/awx/public/static/favicon.ico;\n        }\n\n        location /websocket {\n            # Pass request to the upstream alias\n            proxy_pass http://daphne;\n            # Require http version 1.1 to allow for upgrade requests\n            proxy_http_version 1.1;\n            # We want proxy_buffering off for proxying to websockets.\n            proxy_buffering off;\n            # http://en.wikipedia.org/wiki/X-Forwarded-For\n            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n            # enable this if you use HTTPS:\n            proxy_set_header X-Forwarded-Proto https;\n            # pass the Host: header from the client for the sake of redirects\n            proxy_set_header Host $http_host;\n            # We've set the Host header, so we don't need Nginx to muddle\n            # about with redirects\n            proxy_redirect off;\n            # Depending on the request value, set the Upgrade and\n            # connection headers\n            proxy_set_header Upgrade $http_upgrade;\n            proxy_set_header Connection $connection_upgrade;\n        }\n\n        location / {\n            # Add trailing / if missing\n            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;\n            uwsgi_read_timeout 120s;\n            uwsgi_pass uwsgi;\n            include /etc/nginx/uwsgi_params;                proxy_set_header X-Forwarded-Port 443;\n        }\n    }\n}\n", "settings": "import os\nimport socket\n\ndef get_secret():\n    if os.path.exists(\"/etc/tower/SECRET_KEY\"):\n        return open('/etc/tower/SECRET_KEY', 'rb').read().strip()\n\nADMINS = ()\nSTATIC_ROOT = '/var/lib/awx/public/static'\nPROJECTS_ROOT = '/var/lib/awx/projects'\nJOBOUTPUT_ROOT = '/var/lib/awx/job_status'\n\nSECRET_KEY = get_secret()\n\nALLOWED_HOSTS = ['*']\n\nINTERNAL_API_URL = 'http://127.0.0.1:8052'\n\n# Container environments don't like chroots\nAWX_PROOT_ENABLED = False\n\n# Automatically deprovision pods that go offline\nAWX_AUTO_DEPROVISION_INSTANCES = True\n\nCLUSTER_HOST_ID = socket.gethostname()\nSYSTEM_UUID = '00000000-0000-0000-0000-000000000000'\n\nCSRF_COOKIE_SECURE = False\nSESSION_COOKIE_SECURE = False\n\nSERVER_EMAIL = 'root@localhost'\nDEFAULT_FROM_EMAIL = 'webmaster@localhost'\nEMAIL_SUBJECT_PREFIX = '[AWX] '\n\nEMAIL_HOST = 'localhost'\nEMAIL_PORT = 25\nEMAIL_HOST_USER = ''\nEMAIL_HOST_PASSWORD = ''\nEMAIL_USE_TLS = False\n\nLOGGING['handlers']['console'] = {\n    '()': 'logging.StreamHandler',\n    'level': 'DEBUG',\n    'formatter': 'simple',\n}\n\nLOGGING['loggers']['django.request']['handlers'] = ['console']\nLOGGING['loggers']['rest_framework.request']['handlers'] = ['console']\nLOGGING['loggers']['awx']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.commands.run_callback_receiver']['handlers'] = ['console']\nLOGGING['loggers']['awx.main.tasks']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.scheduler']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['django_auth_ldap']['handlers'] = ['console']\nLOGGING['loggers']['social']['handlers'] = ['console']\nLOGGING['loggers']['system_tracking_migrations']['handlers'] = ['console']\nLOGGING['loggers']['rbac_migrations']['handlers'] = ['console']\nLOGGING['loggers']['awx.isolated.manager.playbooks']['handlers'] = ['console']\nLOGGING['handlers']['callback_receiver'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['task_system'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['tower_warnings'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['rbac_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['system_tracking_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['management_playbooks'] = {'class': 'logging.NullHandler'}\n\nDATABASES = {\n    'default': {\n        'ATOMIC_REQUESTS': True,\n        'ENGINE': 'awx.main.db.profiled_pg',\n        'NAME': 'awx',\n        'USER': 'awx',\n        'PASSWORD': 'awxpass',\n        'HOST': 'awx-postgres.awx.svc.cluster.local',\n        'PORT': '5432',\n    }\n}\n\nif os.getenv(\"DATABASE_SSLMODE\", False):\n    DATABASES['default']['OPTIONS'] = {'sslmode': os.getenv(\"DATABASE_SSLMODE\")}\n\nCACHES = {\n    'default': {\n        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',\n        'LOCATION': '{}:{}'.format(\"awx-memcached.awx.svc.cluster.local\", \"11211\")\n    },\n    'ephemeral': {\n        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',\n    },\n}\n\nBROKER_URL = 'amqp://{}:{}@{}:{}/{}'.format(\n    'guest',\n    'guest',\n    'awx-rabbitmq.awx.svc.cluster.local',\n    '5672',\n    'awx')\n\nCHANNEL_LAYERS = {\n    'default': {'BACKEND': 'asgi_amqp.AMQPChannelLayer',\n                'ROUTING': 'awx.main.routing.channel_routing',\n                'CONFIG': {'url': BROKER_URL}}\n}\n\nUSE_X_FORWARDED_PORT = True\n"}, "kind": "ConfigMap", "metadata": {"creationTimestamp": "2020-03-16T10:18:28Z", "labels": {"app": "tower"}, "name": "awx-tower-configmap", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958144", "selfLink": "/api/v1/namespaces/awx/configmaps/awx-tower-configmap", "uid": "399ccb43-8c32-481d-aba3-617c3395d8cf"}}}
Path": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}, {"mountPath": "/etc/nginx/nginx.conf", "name": "nginx-conf", "readOnly": true, "subPath": "nginx.conf"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}, {"configMap": {"defaultMode": 420, "items": [{"key": "nginx_conf", "path": "nginx.conf"}], "name": "awx-tower-configmap"}, "name": "nginx-conf"}]}}}, "status": {}}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "labels": {"app": "tower"}, "name": "awx-service", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958173", "selfLink": "/api/v1/namespaces/awx/services/awx-service", "uid": "9c2c6278-f50b-4513-bbba-b83ff1abdd6e"}, "spec": {"clusterIP": "10.152.183.136", "ports": [{"port": 80, "protocol": "TCP", "targetPort": 8052}], "selector": {"app": "tower"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}, {"changed": true, "method": "create", "result":changed: [localhost] => (item=tower_web.yaml.j2) => {"ansible_loop_var": "item", "changed": true, "item": "tower_web.yaml.j2", "result": {"results": [{"changed": true, "method": "create", "result": {"apiVersion": "v1", "data": {"admin_password": "cHdBbjgyIXNp", "secret_key": "YXd4c2VjcmV0"}, "kind": "Secret", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "name": "awx-tower-secret", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958158", "selfLink": "/api/v1/namespaces/awx/secrets/awx-tower-secret", "uid": "8280226f-724f-40b3-b31f-d30ef427e107"}, "type": "Opaque"}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "generation": 1, "labels": {"app": "tower"}, "name": "awx-tower-web", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958162", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-tower-web", "uid": "645f9429-d68b-4a7c-9f03-fe8a268197da"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower"}}, "spec": {"containers": [{"image": "ansible/awx_web:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower", "ports": [{"containerPort": 8052, "protocol": "TCP"}], "resources": {"requests": {"cpu": "1", "memory": "2Gi"}}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}, {"mountPath": "/etc/nginx/nginx.conf", "name": "nginx-conf", "readOnly": true, "subPath": "nginx.conf"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}, {"configMap": {"defaultMode": 420, "items": [{"key": "nginx_conf", "path": "nginx.conf"}], "name": "awx-tower-configmap"}, "name": "nginx-conf"}]}}}, "status": {}}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "labels": {"app": "tower"}, "name": "awx-service", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958173", "selfLink": "/api/v1/namespaces/awx/services/awx-service", "uid": "9c2c6278-f50b-4513-bbba-b83ff1abdd6e"}, "spec": {"clusterIP": "10.152.183.136", "ports": [{"port": 80, "protocol": "TCP", "targetPort": 8052}], "selector": {"app": "tower"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}, {"changed": true, "method": "create", "result": {"apiVersion": "extensions/v1beta1", "kind": "Ingress", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "generation": 1, "name": "awx-ingress", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958178", "selfLink": "/apis/extensions/v1beta1/namespaces/awx/ingresses/awx-ingress", "uid": "800855e4-1c8e-4aac-9f0e-61a4a7ec6d50"}, "spec": {"rules": [{"host": "minikube", "http": {"paths": [{"backend": {"serviceName": "awx-service", "servicePort": 80}, "path": "/"}]}}]}, "status": {"loadBalancer": {}}}, "warnings": []}]}}
changed: [localhost] => (item=tower_task.yaml.j2) => {"ansible_loop_var": "item", "changed": true, "item": "tower_task.yaml.j2", "method": "create", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"creationTimestamp": "2020-03-16T10:18:31Z", "generation": 1, "labels": {"app": "tower-task"}, "name": "awx-tower-task", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958183", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-tower-task", "uid": "92ec8294-39a8-4e85-b6c8-fcb02c0ec696"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-task"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-task"}}, "spec": {"containers": [{"command": ["/usr/bin/launch_awx_task.sh"], "envFrom": [{"configMapRef": {"name": "awx-tower-configmap"}}, {"secretRef": {"name": "awx-tower-secret"}}], "image": "ansible/awx_task:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower-task", "resources": {"requests": {"cpu": "500m", "memory": "1Gi"}}, "securityContext": {"privileged": true}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}]}}}, "status": {}}}

TASK [tower : Get the Tower pod information.] **********************************
task path: /opt/ansible/roles/tower/tasks/main.yml:14
[DEPRECATION WARNING]: The 'k8s_facts' module has been renamed to 'k8s_info'. 
This feature will be removed in version 2.13. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost] => {"attempts": 1, "changed": false, "resources": [{"apiVersion": "v1", "kind": "Pod", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "generateName": "awx-tower-web-5b7bc9f756-", "labels": {"app": "tower", "pod-template-hash": "5b7bc9f756"}, "name": "awx-tower-web-5b7bc9f756-8zlhh", "namespace": "awx", "ownerReferences": [{"apiVersion": "apps/v1", "blockOwnerDeletion": true, "controller": true, "kind": "ReplicaSet", "name": "awx-tower-web-5b7bc9f756", "uid": "9d7b8bb7-ea82-48aa-8e40-6658079079aa"}], "resourceVersion": "958200", "selfLink": "/api/v1/namespaces/awx/pods/awx-tower-web-5b7bc9f756-8zlhh", "uid": "820269a4-1534-4782-9a71-2e60a6b8ff43"}, "spec": {"containers": [{"image": "ansible/awx_web:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower", "ports": [{"containerPort": 8052, "protocol": "TCP"}], "resources": {"requests": {"cpu": "1", "memory": "2Gi"}}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}, {"mountPath": "/etc/nginx/nginx.conf", "name": "nginx-conf", "readOnly": true, "subPath": "nginx.conf"}, {"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount", "name": "default-token-sd5hr", "readOnly": true}]}], "dnsPolicy": "ClusterFirst", "enableServiceLinks": true, "nodeName": "minikube", "priority": 0, "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "default", "serviceAccountName": "default", "terminationGracePeriodSeconds": 30, "tolerations": [{"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists", "tolerationSeconds": 300}, {"effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists", "tolerationSeconds": 300}], "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}, {"configMap": {"defaultMode": 420, "items": [{"key": "nginx_conf", "path": "nginx.conf"}], "name": "awx-tower-configmap"}, "name": "nginx-conf"}, {"name": "default-token-sd5hr", "secret": {"defaultMode": 420, "secretName": "default-token-sd5hr"}}]}, "status": {"conditions": [{"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:30Z", "status": "True", "type": "Initialized"}, {"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:32Z", "status": "True", "type": "Ready"}, {"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:32Z", "status": "True", "type": "ContainersReady"}, {"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:30Z", "status": "True", "type": "PodScheduled"}], "containerStatuses": [{"containerID": "containerd://026f570dcda574d82952333914bcf189578830892cbe472fc1829b230cb597ad", "image": "docker.io/ansible/awx_web:9.2.0", "imageID": "docker.io/ansible/awx_web@sha256:57232e6820eb1bfad3c6910bd993e1a8c0d644c0c0fcfb75a25a0b51b3d6fdec", "lastState": {}, "name": "tower", "ready": true, "restartCount": 0, "started": true, "state": {"running": {"startedAt": "2020-03-16T10:18:32Z"}}}], "hostIP": "192.168.0.121", "phase": "Running", "podIP": "10.1.90.166", "podIPs": [{"ip": "10.1.90.166"}], "qosClass": "Burstable", "startTime": "2020-03-16T10:18:30Z"}}]}

TASK [tower : Set the tower pod name as a variable.] ***************************
task path: /opt/ansible/roles/tower/tasks/main.yml:26
ok: [localhost] => {"ansible_facts": {"tower_pod_name": "awx-tower-web-5b7bc9f756-8zlhh"}, "changed": false}

TASK [tower : Verify tower_pod_name is populated.] *****************************
task path: /opt/ansible/roles/tower/tasks/main.yml:30
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [tower : Check if database is populated (auth_user table exists).] ********
task path: /opt/ansible/roles/tower/tasks/main.yml:35
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [tower : Migrate the database if the K8s resources were updated.] *********
task path: /opt/ansible/roles/tower/tasks/main.yml:47
052 DEBUG    awx.main.models.credential adding Microsoft Azure Resource Manager credential type\n2020-03-16 10:21:06,064 DEBUG    awx.main.models.credential adding GitHub Personal Access Token credential type\n2020-03-16 10:21:06,075 DEBUG    awx.main.models.credential adding GitLab Personal Access Token credential type\n2020-03-16 10:21:06,085 DEBUG    awx.main.models.credential adding Insights credential type\n2020-03-16 10:21:06,095 DEBUG    awx.main.models.credential adding Red Hat Virtualization credential type\n2020-03-16 10:21:06,106 DEBUG    awx.main.models.credential adding Ansible Tower credential type\n2020-03-16 10:21:06,119 DEBUG    awx.main.models.credential adding OpenShift or Kubernetes API Bearer Token credential type\n2020-03-16 10:21:06,129 DEBUG    awx.main.models.credential adding CyberArk AIM Central Credential Provider Lookup credential type\n2020-03-16 10:21:06,140 DEBUG    awx.main.models.credential adding Microsoft Azure Key Vault credential type\n2020-03-16 10:21:06,149 DEBUG    awx.main.models.credential adding CyberArk Conjur Secret Lookup credential type\n2020-03-16 10:21:06,161 DEBUG    awx.main.models.credential adding HashiCorp Vault Secret Lookup credential type\n2020-03-16 10:21:06,172 DEBUG    awx.main.models.credential adding HashiCorp Vault Signed SSH credential type\n2020-03-16 10:21:09,414 DEBUG    awx.main.migrations Migrating inventory instance_id for gce to gce_id", "stderr_lines": ["2020-03-16 10:20:46,566 INFO     rbac_migrations Computing role roots..", "2020-03-16 10:20:46,568 INFO     rbac_migrations Found 0 roots in 0.000233 seconds, rebuilding ancestry map", "2020-03-16 10:20:46,568 INFO     rbac_migrations Rebuild completed in 0.000007 seconds", "2020-03-16 10:20:46,568 INFO     rbac_migrations Done.", "2020-03-16 10:20:50,062 INFO     rbac_migrations Computing role roots..", "2020-03-16 10:20:50,064 INFO     rbac_migrations Found 0 roots in 0.000196 seconds, rebuilding ancestry map", "2020-03-16 10:20:50,065 INFO     rbac_migrations Rebuild completed in 0.000008 seconds", "2020-03-16 10:20:50,065 INFO     rbac_migrations Done.", "2020-03-16 10:21:05,926 DEBUG    awx.main.models.credential adding Machine credential type", "2020-03-16 10:21:05,948 DEBUG    awx.main.models.credential adding Source Control credential type", "2020-03-16 10:21:05,959 DEBUG    awx.main.models.credential adding Vault credential type", "2020-03-16 10:21:05,972 DEBUG    awx.main.models.credential adding Network credential type", "2020-03-16 10:21:05,984 DEBUG    awx.main.models.credential adding Amazon Web Services credential type", "2020-03-16 10:21:05,996 DEBUG    awx.main.models.credential adding OpenStack credential type", "2020-03-16 10:21:06,008 DEBUG    awx.main.models.credential adding VMware vCenter credential type", "2020-03-16 10:21:06,018 DEBUG    awx.main.models.credential adding Red Hat Satellite 6 credential type", "2020-03-16 10:21:06,029 DEBUG    awx.main.models.credential adding Red Hat CloudForms credential type", "2020-03-16 10:21:06,040 DEBUG    awx.main.models.credential adding Google Compute Engine credential type", "2020-03-16 10:21:06,052 DEBUG    awx.main.models.credential adding Microsoft Azure Resource Manager credential type", "2020-03-16 10:21:06,064 DEBUG    awx.main.models.credential adding GitHub Personal Access Token credential type", "2020-03-16 10:21:06,075 DEBUG    awx.main.models.credential adding GitLab Personal Access Token credential type", "2020-03-16 10:21:06,085 DEBUG    awx.main.models.credential adding Insights credential type", "2020-03-16 10:21:06,095 DEBUG    awx.main.models.credential adding Red Hat Virtualization credential type", "2020-03-16 10:21:06,106 DEBUG    awx.main.models.credential adding Ansible Tower credential type", "2020-03-16 10:21:06,119 DEBUG    awx.main.models.credential adding OpenShift or Kubernetes API Bearer Token credential type", "2020-03-16 10:21:06,129 DEBUG    awx.main.models.credential adding CyberArk AIM Central Credential Provider Lookup credential type", "2020-03-16 10:21:06,140 DEBUG    awx.main.models.credential adding Microsoft Azure Key Vault credential type", "2020-03-16 10:21:06,149 DEBUG    awx.main.models.credential adding CyberArk Conjur Secret Lookup credential type", "2020-03-16 10:21:06,161 DEBUG    awx.main.models.credential adding HashiCorp Vault Secret Lookup credential type", "2020-03-16 10:21:06,172 DEBUG    awx.main.models.credential adding HashiCorp Vault Signed SSH credential type", "2020-03-16 10:21:09,414 DEBUG    awx.main.migrations Migrating inventory instance_id for gce to gce_id"], "stdout": "Operations to perform:\n  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations:\n  Applying contenttypes.0001_initial... OK\n  Applying contenttypes.0002_remove_content_type_name... OK\n  Applying taggit.0001_initial... OK\n  Applying taggit.0002_auto_20150616_2121... OK\n  Applying auth.0001_initial... OK\n  Applying main.0001_initial... OK\n  Applying main.0002_squashed_v300_release... OK\n  Applying main.0003_squashed_v300_v303_updates... OK\n  Applying main.0004_squashed_v310_release... OK\n  Applying conf.0001_initial... OK\n  Applying conf.0002_v310_copy_tower_settings... OK\n  Applying main.0005_squashed_v310_v313_updates... OK\n  Applying main.0006_v320_release... OK\n  Applying main.0007_v320_data_migrations... OK\n  Applying main.0008_v320_drop_v1_credential_fields... OK\n  Applying main.0009_v322_add_setting_field_for_activity_stream... OK\n  Applying main.0010_v322_add_ovirt4_tower_inventory... OK\n  Applying main.0011_v322_encrypt_survey_passwords... OK\n  Applying main.0012_v322_update_cred_types... OK\n  Applying main.0013_v330_multi_credential... OK\n  Applying auth.0002_alter_permission_name_max_length... OK\n  Applying auth.0003_alter_user_email_max_length... OK\n  Applying auth.0004_alter_user_username_opts... OK\n  Applying auth.0005_alter_user_last_login_null... OK\n  Applying auth.0006_require_contenttypes_0002... OK\n  Applying auth.0007_alter_validators_add_error_messages... OK\n  Applying auth.0008_alter_user_username_max_length... OK\n  Applying auth.0009_alter_user_last_name_max_length... OK\n  Applying auth.0010_alter_group_name_max_length... OK\n  Applying auth.0011_update_proxy_permissions... OK\n  Applying conf.0003_v310_JSONField_changes... OK\n  Applying conf.0004_v320_reencrypt... OK\n  Applying conf.0005_v330_rename_two_session_settings... OK\n  Applying conf.0006_v331_ldap_group_type... OK\n  Applying sessions.0001_initial... OK\n  Applying main.0014_v330_saved_launchtime_configs... OK\n  Applying main.0015_v330_blank_start_args... OK\n  Applying main.0016_v330_non_blank_workflow... OK\n  Applying main.0017_v330_move_deprecated_stdout... OK\n  Applying main.0018_v330_add_additional_stdout_events... OK\n  Applying main.0019_v330_custom_virtualenv... OK\n  Applying main.0020_v330_instancegroup_policies... OK\n  Applying main.0021_v330_declare_new_rbac_roles... OK\n  Applying main.0022_v330_create_new_rbac_roles... OK\n  Applying main.0023_v330_inventory_multicred... OK\n  Applying main.0024_v330_create_user_session_membership... OK\n  Applying main.0025_v330_add_oauth_activity_stream_registrar... OK\n  Applying oauth2_provider.0001_initial... OK\n  Applying main.0026_v330_delete_authtoken... OK\n  Applying main.0027_v330_emitted_events... OK\n  Applying main.0028_v330_add_tower_verify... OK\n  Applying main.0030_v330_modify_application... OK\n  Applying main.0031_v330_encrypt_oauth2_secret... OK\n  Applying main.0032_v330_polymorphic_delete... OK\n  Applying main.0033_v330_oauth_help_text... OK\n  Applying main.0034_v330_delete_user_role... OK\n  Applying main.0035_v330_more_oauth2_help_text... OK\n  Applying main.0036_v330_credtype_remove_become_methods... OK\n  Applying main.0037_v330_remove_legacy_fact_cleanup... OK\n  Applying main.0038_v330_add_deleted_activitystream_actor... OK\n  Applying main.0039_v330_custom_venv_help_text... OK\n  Applying main.0040_v330_unifiedjob_controller_node... OK\n  Applying main.0041_v330_update_oauth_refreshtoken... OK\n  Applying main.0042_v330_org_member_role_deparent... OK\n  Applying main.0043_v330_oauth2accesstoken_modified... OK\n  Applying main.0044_v330_add_inventory_update_inventory... OK\n  Applying main.0045_v330_instance_managed_by_policy... OK\n  Applying main.0046_v330_remove_client_credentials_grant... OK\n  Applying main.0047_v330_activitystream_instance... OK\n  Applying main.0048_v330_django_created_modified_by_model_name... OK\n  Applying main.0049_v330_validate_instance_capacity_adjustment... OK\n  Applying main.0050_v340_drop_celery_tables... OK\n  Applying main.0051_v340_job_slicing... OK\n  Applying main.0052_v340_remove_project_scm_delete_on_next_update... OK\n  Applying main.0053_v340_workflow_inventory... OK\n  Applying main.0054_v340_workflow_convergence... OK\n  Applying main.0055_v340_add_grafana_notification... OK\n  Applying main.0056_v350_custom_venv_history... OK\n  Applying main.0057_v350_remove_become_method_type... OK\n  Applying main.0058_v350_remove_limit_limit... OK\n  Applying main.0059_v350_remove_adhoc_limit... OK\n  Applying main.0060_v350_update_schedule_uniqueness_constraint... OK\n  Applying main.0061_v350_track_native_credentialtype_source... OK\n  Applying main.0062_v350_new_playbook_stats... OK\n  Applying main.0063_v350_org_host_limits... OK\n  Applying main.0064_v350_analytics_state... OK\n  Applying main.0065_v350_index_job_status... OK\n  Applying main.0066_v350_inventorysource_custom_virtualenv... OK\n  Applying main.0067_v350_credential_plugins... OK\n  Applying main.0068_v350_index_event_created... OK\n  Applying main.0069_v350_generate_unique_install_uuid... OK\n  Applying main.0070_v350_gce_instance_id... OK\n  Applying main.0071_v350_remove_system_tracking... OK\n  Applying main.0072_v350_deprecate_fields... OK\n  Applying main.0073_v360_create_instance_group_m2m... OK\n  Applying main.0074_v360_migrate_instance_group_relations... OK\n  Applying main.0075_v360_remove_old_instance_group_relations... OK\n  Applying main.0076_v360_add_new_instance_group_relations... OK\n  Applying main.0077_v360_add_default_orderings... OK\n  Applying main.0078_v360_clear_sessions_tokens_jt... OK\n  Applying main.0079_v360_rm_implicit_oauth2_apps... OK\n  Applying main.0080_v360_replace_job_origin... OK\n  Applying main.0081_v360_notify_on_start... OK\n  Applying main.0082_v360_webhook_http_method... OK\n  Applying main.0083_v360_job_branch_override... OK\n  Applying main.0084_v360_token_description... OK\n  Applying main.0085_v360_add_notificationtemplate_messages... OK\n  Applying main.0086_v360_workflow_approval... OK\n  Applying main.0087_v360_update_credential_injector_help_text... OK\n  Applying main.0088_v360_dashboard_optimizations... OK\n  Applying main.0089_v360_new_job_event_types... OK\n  Applying main.0090_v360_WFJT_prompts... OK\n  Applying main.0091_v360_approval_node_notifications... OK\n  Applying main.0092_v360_webhook_mixin... OK\n  Applying main.0093_v360_personal_access_tokens... OK\n  Applying main.0094_v360_webhook_mixin2... OK\n  Applying main.0095_v360_increase_instance_version_length... OK\n  Applying main.0096_v360_container_groups... OK\n  Applying main.0097_v360_workflowapproval_approved_or_denied_by... OK\n  Applying main.0098_v360_rename_cyberark_aim_credential_type... OK\n  Applying main.0099_v361_license_cleanup... OK\n  Applying main.0100_v370_projectupdate_job_tags... OK\n  Applying main.0101_v370_generate_new_uuids_for_iso_nodes... OK\n  Applying main.0102_v370_unifiedjob_canceled... OK\n  Applying main.0103_v370_remove_computed_fields... OK\n  Applying main.0104_v370_cleanup_old_scan_jts... OK\n  Applying main.0105_v370_remove_jobevent_parent_and_hosts... OK\n  Applying main.0106_v370_remove_inventory_groups_with_active_failures... OK\n  Applying main.0107_v370_workflow_convergence_api_toggle... OK\n  Applying main.0108_v370_unifiedjob_dependencies_processed... OK\n  Applying oauth2_provider.0002_08_updates... OK\n  Applying oauth2_provider.0003_auto_20160316_1503... OK\n  Applying oauth2_provider.0004_auto_20160525_1623... OK\n  Applying oauth2_provider.0005_auto_20170514_1141... OK\n  Applying oauth2_provider.0006_auto_20171214_2232... OK\n  Applying sites.0001_initial... OK\n  Applying sites.0002_alter_domain_unique... OK\n  Applying social_django.0001_initial... OK\n  Applying social_django.0002_add_related_name... OK\n  Applying social_django.0003_alter_email_max_length... OK\n  Applying social_django.0004_auto_20160423_0400... OK\n  Applying social_django.0005_auto_20160727_2333... OK\n  Applying social_django.0006_partial... OK\n  Applying social_django.0007_code_timestamp... OK\n  Applying social_django.0008_partial_timestamp... OK\n  Applying sso.0001_initial... OK\n  Applying sso.0002_expand_provider_options... OK\n  Applying taggit.0003_taggeditem_add_unique_index... OK", "stdout_lines": ["Operations to perform:", "  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", "  Applying contenttypes.0001_initial... OK", "  Applying contenttypes.0002_remove_content_type_name... OK", "  Applying taggit.0001_initial... OK", "  Applying taggit.0002_auto_20150616_2121... OK", "  Applying auth.0001_initial... OK", "  Applying main.0001_initial... OK", "  Applying main.0002_squashed_v300_release... OK", "  Applying main.0003_squashed_v300_v303_updates... OK", "  Applying main.0004_squashed_v310_release... OK", "  Applying conf.0001_initial... OK", "  Applying conf.0002_v310_copy_tower_settings... OK", "  Applying main.0005_squashed_v310_v313_updates... OK", "  Applying main.0006_v320_release... OK", "  Applying main.0007_v320_data_migrations... OK", "  Applying main.0008_v320_drop_v1_credential_fields... OK", "  Applying main.0009_v322_add_setting_field_for_activity_stream... OK", "  Applying main.0010_v322_add_ovirt4_tower_inventory... OK", "  Applying main.0011_v322_encrypt_survey_passwords... OK", "  Applying main.0012_v322_update_cred_types... OK", "  Applying main.0013_v330_multi_credential... OK", "  Applying auth.0002_alter_permission_name_max_length... OK", "  Applying auth.0003_alter_user_email_max_length... OK", "  Applying auth.0004_alter_user_username_opts... OK", "  Applying auth.0005_alter_user_last_login_null... OK", "  Applying auth.0006_require_contenttypes_0002... OK", "  Applying auth.0007_alter_validators_add_error_messages... OK", "  Applying auth.0008_alter_user_username_max_length... OK", "  Applying auth.0009_alter_user_last_name_max_length... OK", "  Applying auth.0010_alter_group_name_max_length... OK", "  Applying auth.0011_update_proxy_permissions... OK", "  Applying conf.0003_v310_JSONField_changes... OK", "  Applying conf.0004_v320_reencrypt... OK", "  Applying conf.0005_v330_rename_two_session_settings... OK", "  Applying conf.0006_v331_ldap_group_type... OK", "  Applying sessions.0001_initial... OK", "  Applying main.0014_v330_saved_launchtime_configs... OK", "  Applying main.0015_v330_blank_start_args... OK", "  Applying main.0016_v330_non_blank_workflow... OK", "  Applying main.0017_v330_move_deprecated_stdout... OK", "  Applying main.0018_v330_add_additional_stdout_events... OK", "  Applying main.0019_v330_custom_virtualenv... OK", "  Applying main.0020_v330_instancegroup_policies... OK", "  Applying main.0021_v330_declare_new_rbac_roles... OK", "  Applying main.0022_v330_create_new_rbac_roles... OK", "  Applying main.0023_v330_inventory_multicred... OK", "  Applying main.0024_v330_create_user_session_membership... OK", "  Applying main.0025_v330_add_oauth_activity_stream_registrar... OK", "  Applying oauth2_provider.0001_initial... OK", "  Applying main.0026_v330_delete_authtoken... OK", "  Applying main.0027_v330_emitted_events... OK", "  Applying main.0028_v330_add_tower_verify... OK", "  Applying main.0030_v330_modify_application... OK", "  Applying main.0031_v330_encrypt_oauth2_secret... OK", "  Applying main.0032_v330_polymorphic_delete... OK", "  Applying main.0033_v330_oauth_help_text... OK", "  Applying main.0034_v330_delete_user_role... OK", "  Applying main.0035_v330_more_oauth2_help_text... OK", "  Applying main.0036_v330_credtype_remove_become_methods... OK", "  Applying main.0037_v330_remove_legacy_fact_cleanup... OK", "  Applying main.0038_v330_add_deleted_activitystream_actor... OK", "  Applying main.0039_v330_custom_venv_help_text... OK", "  Applying main.0040_v330_unifiedjob_controller_node... OK", "  Applying main.0041_v330_update_oauth_refreshtoken... OK", "  Applying main.0042_v330_org_member_role_deparent... OK", "  Applying main.0043_v330_oauth2accesstoken_modified... OK", "  Applying main.0044_v330_add_inventory_update_inventory... OK", "  Applying main.0045_v330_instance_managed_by_policy... OK", "  Applying main.0046_v330_remove_client_credentials_grant... OK", "  Applying main.0047_v330_activitystream_instance... OK", "  Applying main.0048_v330_django_created_modified_by_model_name... OK", "  Applying main.0049_v330_validate_instance_capacity_adjustment... OK", "  Applying main.0050_v340_drop_celery_tables... OK", "  Applying main.0051_v340_job_slicing... OK", "  Applying main.0052_v340_remove_project_scm_delete_on_next_update... OK", "  Applying main.0053_v340_workflow_inventory... OK", "  Applying main.0054_v340_workflow_convergence... OK", "  Applying main.0055_v340_add_grafana_notification... OK", "  Applying main.0056_v350_custom_venv_history... OK", "  Applying main.0057_v350_remove_become_method_type... OK", "  Applying main.0058_v350_remove_limit_limit... OK", "  Applying main.0059_v350_remove_adhoc_limit... OK", "  Applying main.0060_v350_update_schedule_uniqueness_constraint... OK", "  Applying main.0061_v350_track_native_credentialtype_source... OK", "  Applying main.0062_v350_new_playbook_stats... OK", "  Applying main.0063_v350_org_host_limits... OK", "  Applying main.0064_v350_analytics_state... OK", "  Applying main.0065_v350_index_job_status... OK", "  Applying main.0066_v350_inventorysource_custom_virtualenv... OK", "  Applying main.0067_v350_credential_plugins... OK", "  Applying main.0068_v350_index_event_created... OK", "  Applying main.0069_v350_generate_unique_install_uuid... OK", "  Applying main.0070_v350_gce_instance_id... OK", "  Applying main.0071_v350_remove_system_tracking... OK", "  Applying main.0072_v350_deprecate_fields... OK", "  Applying main.0073_v360_create_instance_group_m2m... OK", "  Applying main.0074_v360_migrate_instance_group_relations... OK", "  Applying main.0075_v360_remove_old_instance_group_relations... OK", "  Applying main.0076_v360_add_new_instance_group_relations... OK", "  Applying main.0077_v360_add_default_orderings... OK", "  Applying main.0078_v360_clear_sessions_tokens_jt... OK", "  Applying main.0079_v360_rm_implicit_oauth2_apps... OK", "  Applying main.0080_v360_replace_job_origin... OK", "  Applying main.0081_v360_notify_on_start... OK", "  Applying main.0082_v360_webhook_http_method... OK", "  Applying main.0083_v360_job_branch_override... OK", "  Applying main.0084_v360_token_description... OK", "  Applying main.0085_v360_add_notificationtemplate_messages... OK", "  Applying main.0086_v360_workflow_approval... OK", "  Applying main.0087_v360_update_credential_injector_help_text... OK", "  Applying main.0088_v360_dashboard_optimizations... OK", "  Applying main.0089_v360_new_job_event_types... OK", "  Applying main.0090_v360_WFJT_prompts... OK", "  Applying main.0091_v360_approval_node_notifications... OK", "  Applying main.0092_v360_webhook_mixin... OK", "  Applying main.0093_v360_personal_access_tokens... OK", "  Applying main.0094_v360_webhook_mixin2... OK", "  Applying main.0095_v360_increase_instance_version_length... OK", "  Applying main.0096_v360_container_groups... OK", "  Applying main.0097_v360_workflowapproval_approved_or_denied_by... OK", "  Applying mchanged: [localhost] => {"changed": true, "cmd": "kubectl exec -n awx awx-tower-web-5b7bc9f756-8zlhh -- bash -c \"awx-manage migrate --noinput\"", "delta": "0:03:22.003322", "end": "2020-03-16 10:22:01.655453", "rc": 0, "start": "2020-03-16 10:18:39.652131", "stderr": "2020-03-16 10:20:46,566 INFO     rbac_migrations Computing role roots..\n2020-03-16 10:20:46,568 INFO     rbac_migrations Found 0 roots in 0.000233 seconds, rebuilding ancestry map\n2020-03-16 10:20:46,568 INFO     rbac_migrations Rebuild completed in 0.000007 seconds\n2020-03-16 10:20:46,568 INFO     rbac_migrations Done.\n2020-03-16 10:20:50,062 INFO     rbac_migrations Computing role roots..\n2020-03-16 10:20:50,064 INFO     rbac_migrations Found 0 roots in 0.000196 seconds, rebuilding ancestry map\n2020-03-16 10:20:50,065 INFO     rbac_migrations Rebuild completed in 0.000008 seconds\n2020-03-16 10:20:50,065 INFO     rbac_migrations Done.\n2020-03-16 10:21:05,926 DEBUG    awx.main.models.credential adding Machine credential type\n2020-03-16 10:21:05,948 DEBUG    awx.main.models.credential adding Source Control credential type\n2020-03-16 10:21:05,959 DEBUG    awx.main.models.credential adding Vault credential type\n2020-03-16 10:21:05,972 DEBUG    awx.main.models.credential adding Network credential type\n2020-03-16 10:21:05,984 DEBUG    awx.main.models.credential adding Amazon Web Services credential type\n2020-03-16 10:21:05,996 DEBUG    awx.main.models.credential adding OpenStack credential type\n2020-03-16 10:21:06,008 DEBUG    awx.main.models.credential adding VMware vCenter credential type\n2020-03-16 10:21:06,018 DEBUG    awx.main.models.credential adding Red Hat Satellite 6 credential type\n2020-03-16 10:21:06,029 DEBUG    awx.main.models.credential adding Red Hat CloudForms credential type\n2020-03-16 10:21:06,040 DEBUG    awx.main.models.credential adding Google Compute Engine credential type\n2020-03-16 10:21:06,052 DEBUG    awx.main.models.credential adding Microsoft Azure Resource Manager credential type\n2020-03-16 10:21:06,064 DEBUG    awx.main.models.credential adding GitHub Personal Access Token credential type\n2020-03-16 10:21:06,075 DEBUG    awx.main.models.credential adding GitLab Personal Access Token credential type\n2020-03-16 10:21:06,085 DEBUG    awx.main.models.credential adding Insights credential type\n2020-03-16 10:21:06,095 DEBUG    awx.main.models.credential adding Red Hat Virtualization credential type\n2020-03-16 10:21:06,106 DEBUG    awx.main.models.credential adding Ansible Tower credential type\n2020-03-16 10:21:06,119 DEBUG    awx.main.models.credential adding OpenShift or Kubernetes API Bearer Token credential type\n2020-03-16 10:21:06,129 DEBUG    awx.main.models.credential adding CyberArk AIM Central Credential Provider Lookup credential type\n2020-03-16 10:21:06,140 DEBUG    awx.main.models.credential adding Microsoft Azure Key Vault credential type\n2020-03-16 10:21:06,149 DEBUG    awx.main.models.credential adding CyberArk Conjur Secret Lookup credential type\n2020-03-16 10:21:06,161 DEBUG    awx.main.models.credential adding HashiCorp Vault Secret Lookup credential type\n2020-03-16 10:21:06,172 DEBUG    awx.main.models.credential adding HashiCorp Vault Signed SSH credential type\n2020-03-16 10:21:09,414 DEBUG    awx.main.migrations Migrating inventory instance_id for gce to gce_id", "stderr_lines": ["2020-03-16 10:20:46,566 INFO     rbac_migrations Computing role roots..", "2020-03-16 10:20:46,568 INFO     rbac_migrations Found 0 roots in 0.000233 seconds, rebuilding ancestry map", "2020-03-16 10:20:46,568 INFO     rbac_migrations Rebuild completed in 0.000007 seconds", "2020-03-16 10:20:46,568 INFO     rbac_migrations Done.", "2020-03-16 10:20:50,062 INFO     rbac_migrations Computing role roots..", "2020-03-16 10:20:50,064 INFO     rbac_migrations Found 0 roots in 0.000196 seconds, rebuilding ancestry map", "2020-03-16 10:20:50,065 INFO     rbac_migrations Rebuild completed in 0.000008 seconds", "2020-03-16 10:20:50,065 INFO     rbac_migrations Done.", "2020-03-16 10:21:05,926 DEBUG    awx.main.models.credential adding Machine credential type", "2020-03-16 10:21:05,948 DEBUG    awx.main.models.credential adding Source Control credential type", "2020-03-16 10:21:05,959 DEBUG    awx.main.models.credential adding Vault credential type", "2020-03-16 10:21:05,972 DEBUG    awx.main.models.credential adding Network credential type", "2020-03-16 10:21:05,984 DEBUG    awx.main.models.credential adding Amazon Web Services credential type", "2020-03-16 10:21:05,996 DEBUG    awx.main.models.credential adding OpenStack credential type", "2020-03-16 10:21:06,008 DEBUG    awx.main.models.credential adding VMware vCenter credential type", "2020-03-16 10:21:06,018 DEBUG    awx.main.models.credential adding Red Hat Satellite 6 credential type", "2020-03-16 10:21:06,029 DEBUG    awx.main.models.credential adding Red Hat CloudForms credential type", "2020-03-16 10:21:06,040 DEBUG    awx.main.models.credential adding Google Compute Engine credential type", "2020-03-16 10:21:06,052 DEBUG    awx.main.models.credential adding Microsoft Azure Resource Manager credential type", "2020-03-16 10:21:06,064 DEBUG    awx.main.models.credential adding GitHub Personal Access Token credential type", "2020-03-16 10:21:06,075 DEBUG    awx.main.models.credential adding GitLab Personal Access Token credential type", "2020-03-16 10:21:06,085 DEBUG    awx.main.models.credential adding Insights credential type", "2020-03-16 10:21:06,095 DEBUG    awx.main.models.credential adding Red Hat Virtualization credential type", "2020-03-16 10:21:06,106 DEBUG    awx.main.models.credential adding Ansible Tower credential type", "2020-03-16 10:21:06,119 DEBUG    awx.main.models.credential adding OpenShift or Kubernetes API Bearer Token credential type", "2020-03-16 10:21:06,129 DEBUG    awx.main.models.credential adding CyberArk AIM Central Credential Provider Lookup credential type", "2020-03-16 10:21:06,140 DEBUG    awx.main.models.credential adding Microsoft Azure Key Vault credential type", "2020-03-16 10:21:06,149 DEBUG    awx.main.models.credential adding CyberArk Conjur Secret Lookup credential type", "2020-03-16 10:21:06,161 DEBUG    awx.main.models.credential adding HashiCorp Vault Secret Lookup credential type", "2020-03-16 10:21:06,172 DEBUG    awx.main.models.credential adding HashiCorp Vault Signed SSH credential type", "2020-03-16 10:21:09,414 DEBUG    awx.main.migrations Migrating inventory instance_id for gce to gce_id"], "stdout": "Operations to perform:\n  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations:\n  Applying contenttypes.0001_initial... OK\n  Applying contenttypes.0002_remove_content_type_name... OK\n  Applying taggit.0001_initial... OK\n  Applying taggit.0002_auto_20150616_2121... OK\n  Applying auth.0001_initial... OK\n  Applying main.0001_initial... OK\n  Applying main.0002_squashed_v300_release... OK\n  Applying main.0003_squashed_v300_v303_updates... OK\n  Applying main.0004_squashed_v310_release... OK\n  Applying conf.0001_initial... OK\n  Applying conf.0002_v310_copy_tower_settings... OK\n  Applying main.0005_squashed_v310_v313_updates... OK\n  Applying main.0006_v320_release... OK\n  Applying main.0007_v320_data_migrations... OK\n  Applying main.0008_v320_drop_v1_credential_fields... OK\n  Applying main.0009_v322_add_setting_field_for_activity_stream... OK\n  Applying main.0010_v322_add_ovirt4_tower_inventory... OK\n  Applying main.0011_v322_encrypt_survey_passwords... OK\n  Applying main.0012_v322_update_cred_types... OK\n  Applying main.0013_v330_multi_credential... OK\n  Applying auth.0002_alter_permission_name_max_length... OK\n  Applying auth.0003_alter_user_email_max_length... OK\n  Applying auth.0004_alter_user_username_opts... OK\n  Applying auth.0005_alter_user_last_login_null... OK\n  Applying auth.0006_require_contenttypes_0002... OK\n  Applying auth.0007_alter_validators_add_error_messages... OK\n  Applying auth.0008_alter_user_username_max_length... OK\n  Applying auth.0009_alter_user_last_name_max_length... OK\n  Applying auth.0010_alter_group_name_max_length... OK\n  Applying auth.0011_update_proxy_permissions... OK\n  Applying conf.0003_v310_JSONField_changes... OK\n  Applying conf.0004_v320_reencrypt... OK\n  Applying conf.0005_v330_rename_two_session_settings... OK\n  Applying conf.0006_v331_ldap_group_type... OK\n  Applying sessions.0001_initial... OK\n  Applying main.0014_v330_saved_launchtime_configs... OK\n  Applying main.0015_v330_blank_start_args... OK\n  Applying main.0016_v330_non_blank_workflow... OK\n  Applying main.0017_v330_move_deprecated_stdout... OK\n  Applying main.0018_v330_add_additional_stdout_events... OK\n  Applying main.0019_v330_custom_virtualenv... OK\n  Applying main.0020_v330_instancegroup_policies... OK\n  Applying main.0021_v330_declare_new_rbac_roles... OK\n  Applying main.0022_v330_create_new_rbac_roles... OK\n  Applying main.0023_v330_inventory_multicred... OK\n  Applying main.0024_v330_create_user_session_membership... OK\n  Applying main.0025_v330_add_oauth_activity_stream_registrar... OK\n  Applying oauth2_provider.0001_initial... OK\n  Applying main.0026_v330_delete_authtoken... OK\n  Applying main.0027_v330_emitted_events... OK\n  Applying main.0028_v330_add_tower_verify... OK\n  Applying main.0030_v330_modify_application... OK\n  Applying main.0031_v330_encrypt_oauth2_secret... OK\n  Applying main.0032_v330_polymorphic_delete... OK\n  Applying main.0033_v330_oauth_help_text... OK\n  Applying main.0034_v330_delete_user_role... OK\n  Applying main.0035_v330_more_oauth2_help_text... OK\n  Applying main.0036_v330_credtype_remove_become_methods... OK\n  Applying main.0037_v330_remove_legacy_fact_cleanup... OK\n  Applying main.0038_v330_add_deleted_activitystream_actor... OK\n  Applying main.0039_v330_custom_venv_help_text... OK\n  Applying main.0040_v330_unifiedjob_controller_node... OK\n  Applying main.0041_v330_update_oauth_refreshtoken... OK\n  Applying main.0042_v330_org_member_role_deparent... OK\n  Applying main.0043_v330_oauth2accesstoken_modified... OK\n  Applying main.0044_v330_add_inventory_update_inventory... OK\n  Applying main.0045_v330_instance_managed_by_policy... OK\n  Applying main.0046_v330_remove_client_credentials_grant... OK\n  Applying main.0047_v330_activitystream_instance... OK\n  Applying main.0048_v330_django_created_modified_by_model_name... OK\n  Applying main.0049_v330_validate_instance_capacity_adjustment... OK\n  Applying main.0050_v340_drop_celery_tables... OK\n  Applying main.0051_v340_job_slicing... OK\n  Applying main.0052_v340_remove_project_scm_delete_on_next_update... OK\n  Applying main.0053_v340_workflow_inventory... OK\n  Applying main.0054_v340_workflow_convergence... OK\n  Applying main.0055_v340_add_grafana_notification... OK\n  Applying main.0056_v350_custom_venv_history... OK\n  Applying main.0057_v350_remove_become_method_type... OK\n  Applying main.0058_v350_remove_limit_limit... OK\n  Applying main.0059_v350_remove_adhoc_limit... OK\n  Applying main.0060_v350_update_schedule_uniqueness_constraint... OK\n  Applying main.0061_v350_track_native_credentialtype_source... OK\n  Applying main.0062_v350_new_playbook_stats... OK\n  Applying main.0063_v350_org_host_limits... OK\n  Applying main.0064_v350_analytics_state... OK\n  Applying main.0065_v350_index_job_status... OK\n  Applying main.0066_v350_inventorysource_custom_virtualenv... OK\n  Applying main.0067_v350_credential_plugins... OK\n  Applying main.0068_v350_index_event_created... OK\n  Applying main.0069_v350_generate_unique_install_uuid... OK\n  Applying main.0070_v350_gce_instance_id... OK\n  Applying main.0071_v350_remove_system_tracking... OK\n  Applying main.0072_v350_deprecate_fields... OK\n  Applying main.0073_v360_create_instance_group_m2m... OK\n  Applying main.0074_v360_migrate_instance_group_relations... OK\n  Applying main.0075_v360_remove_old_instance_group_relations... OK\n  Applying main.0076_v360_add_new_instance_group_relations... OK\n  Applying main.0077_v360_add_default_orderings... OK\n  Applying main.0078_v360_clear_sessions_tokens_jt... OK\n  Applying main.0079_v360_rm_implicit_oauth2_apps... OK\n  Applying main.0080_v360_replace_job_origin... OK\n  Applying main.0081_v360_notify_on_start... OK\n  Applying main.0082_v360_webhook_http_method... OK\n  Applying main.0083_v360_job_branch_override... OK\n  Applying main.0084_v360_token_description... OK\n  Applying main.0085_v360_add_notificationtemplate_messages... OK\n  Applying main.0086_v360_workflow_approval... OK\n  Applying main.0087_v360_update_credential_injector_help_text... OK\n  Applying main.0088_v360_dashboard_optimizations... OK\n  Applying main.0089_v360_new_job_event_types... OK\n  Applying main.0090_v360_WFJT_prompts... OK\n  Applying main.0091_v360_approval_node_notifications... OK\n  Applying main.0092_v360_webhook_mixin... OK\n  Applying main.0093_v360_personal_access_tokens... OK\n  Applying main.0094_v360_webhook_mixin2... OK\n  Applying main.0095_v360_increase_instance_version_length... OK\n  Applying main.0096_v360_container_groups... OK\n  Applying main.0097_v360_workflowapproval_approved_or_denied_by... OK\n  Applying main.0098_v360_rename_cyberark_aim_credential_type... OK\n  Applying main.0099_v361_license_cleanup... OK\n  Applying main.0100_v370_projectupdate_job_tags... OK\n  Applying main.0101_v370_generate_new_uuids_for_iso_nodes... OK\n  Applying main.0102_v370_unifiedjob_canceled... OK\n  Applying main.0103_v370_remove_computed_fields... OK\n  Applying main.0104_v370_cleanup_old_scan_jts... OK\n  Applying main.0105_v370_remove_jobevent_parent_and_hosts... OK\n  Applying main.0106_v370_remove_inventory_groups_with_active_failures... OK\n  Applying main.0107_v370_workflow_convergence_api_toggle... OK\n  Applying main.0108_v370_unifiedjob_dependencies_processed... OK\n  Applying oauth2_provider.0002_08_updates... OK\n  Applying oauth2_provider.0003_auto_20160316_1503... OK\n  Applying oauth2_provider.0004_auto_20160525_1623... OK\n  Applying oauth2_provider.0005_auto_20170514_1141... OK\n  Applying oauth2_provider.0006_auto_20171214_2232... OK\n  Applying sites.0001_initial... OK\n  Applying sites.0002_alter_domain_unique... OK\n  Applying social_django.0001_initial... OK\n  Applying social_django.0002_add_related_name... OK\n  Applying social_django.0003_alter_email_max_length... OK\n  Applying social_django.0004_auto_20160423_0400... OK\n  Applying social_django.0005_auto_20160727_2333... OK\n  Applying social_django.0006_partial... OK\n  Applying social_django.0007_code_timestamp... OK\n  Applying social_django.0008_partial_timestamp... OK\n  Applying sso.0001_initial... OK\n  Applying sso.0002_expand_provider_options... OK\n  Applying taggit.0003_taggeditem_add_unique_index... OK", "stdout_lines": ["Operations to perform:", "  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", "  Applying contenttypes.0001_initial... OK", "  Applying contenttypes.0002_remove_content_type_name... OK", "  Applying taggit.0001_initial... OK", "  Applying taggit.0002_auto_20150616_2121... OK", "  Applying auth.0001_initial... OK", "  Applying main.0001_initial... OK", "  Applying main.0002_squashed_v300_release... OK", "  Applying main.0003_squashed_v300_v303_updates... OK", "  Applying main.0004_squashed_v310_release... OK", "  Applying conf.0001_initial... OK", "  Applying conf.0002_v310_copy_tower_settings... OK", "  Applying main.0005_squashed_v310_v313_updates... OK", "  Applying main.0006_v320_release... OK", "  Applying main.0007_v320_data_migrations... OK", "  Applying main.0008_v320_drop_v1_credential_fields... OK", "  Applying main.0009_v322_add_setting_field_for_activity_stream... OK", "  Applying main.0010_v322_add_ovirt4_tower_inventory... OK", "  Applying main.0011_v322_encrypt_survey_passwords... OK", "  Applying main.0012_v322_update_cred_types... OK", "  Applying main.0013_v330_multi_credential... OK", "  Applying auth.0002_alter_permission_name_max_length... OK", "  Applying auth.0003_alter_user_email_max_length... OK", "  Applying auth.0004_alter_user_username_opts... OK", "  Applying auth.0005_alter_user_last_login_null... OK", "  Applying auth.0006_require_contenttypes_0002... OK", "  Applying auth.0007_alter_validators_add_error_messages... OK", "  Applying auth.0008_alter_user_username_max_length... OK", "  Applying auth.0009_alter_user_last_name_max_length... OK", "  Applying auth.0010_alter_group_name_max_length... OK", "  Applying auth.0011_update_proxy_permissions... OK", "  Applying conf.0003_v310_JSONField_changes... OK", "  Applying conf.0004_v320_reencrypt... OK", "  Applying conf.0005_v330_rename_two_session_settings... OK", "  Applying conf.0006_v331_ldap_group_type... OK", "  Applying sessions.0001_initial... OK", "  Applying main.0014_v330_saved_launchtime_configs... OK", "  Applying main.0015_v330_blank_start_args... OK", "  Applying main.0016_v330_non_blank_workflow... OK", "  Applying main.0017_v330_move_deprecated_stdout... OK", "  Applying main.0018_v330_add_additional_stdout_events... OK", "  Applying main.0019_v330_custom_virtualenv... OK", "  Applying main.0020_v330_instancegroup_policies... OK", "  Applying main.0021_v330_declare_new_rbac_roles... OK", "  Applying main.0022_v330_create_new_rbac_roles... OK", "  Applying main.0023_v330_inventory_multicred... OK", "  Applying main.0024_v330_create_user_session_membership... OK", "  Applying main.0025_v330_add_oauth_activity_stream_registrar... OK", "  Applying oauth2_provider.0001_initial... OK", "  Applying main.0026_v330_delete_authtoken... OK", "  Applying main.0027_v330_emitted_events... OK", "  Applying main.0028_v330_add_tower_verify... OK", "  Applying main.0030_v330_modify_application... OK", "  Applying main.0031_v330_encrypt_oauth2_secret... OK", "  Applying main.0032_v330_polymorphic_delete... OK", "  Applying main.0033_v330_oauth_help_text... OK", "  Applying main.0034_v330_delete_user_role... OK", "  Applying main.0035_v330_more_oauth2_help_text... OK", "  Applying main.0036_v330_credtype_remove_become_methods... OK", "  Applying main.0037_v330_remove_legacy_fact_cleanup... OK", "  Applying main.0038_v330_add_deleted_activitystream_actor... OK", "  Applying main.0039_v330_custom_venv_help_text... OK", "  Applying main.0040_v330_unifiedjob_controller_node... OK", "  Applying main.0041_v330_update_oauth_refreshtoken... OK", "  Applying main.0042_v330_org_member_role_deparent... OK", "  Applying main.0043_v330_oauth2accesstoken_modified... OK", "  Applying main.0044_v330_add_inventory_update_inventory... OK", "  Applying main.0045_v330_instance_managed_by_policy... OK", "  Applying main.0046_v330_remove_client_credentials_grant... OK", "  Applying main.0047_v330_activitystream_instance... OK", "  Applying main.0048_v330_django_created_modified_by_model_name... OK", "  Applying main.0049_v330_validate_instance_capacity_adjustment... OK", "  Applying main.0050_v340_drop_celery_tables... OK", "  Applying main.0051_v340_job_slicing... OK", "  Applying main.0052_v340_remove_project_scm_delete_on_next_update... OK", "  Applying main.0053_v340_workflow_inventory... OK", "  Applying main.0054_v340_workflow_convergence... OK", "  Applying main.0055_v340_add_grafana_notification... OK", "  Applying main.0056_v350_custom_venv_history... OK", "  Applying main.0057_v350_remove_become_method_type... OK", "  Applying main.0058_v350_remove_limit_limit... OK", "  Applying main.0059_v350_remove_adhoc_limit... OK", "  Applying main.0060_v350_update_schedule_uniqueness_constraint... OK", "  Applying main.0061_v350_track_native_credentialtype_source... OK", "  Applying main.0062_v350_new_playbook_stats... OK", "  Applying main.0063_v350_org_host_limits... OK", "  Applying main.0064_v350_analytics_state... OK", "  Applying main.0065_v350_index_job_status... OK", "  Applying main.0066_v350_inventorysource_custom_virtualenv... OK", "  Applying main.0067_v350_credential_plugins... OK", "  Applying main.0068_v350_index_event_created... OK", "  Applying main.0069_v350_generate_unique_install_uuid... OK", "  Applying main.0070_v350_gce_instance_id... OK", "  Applying main.0071_v350_remove_system_tracking... OK", "  Applying main.0072_v350_deprecate_fields... OK", "  Applying main.0073_v360_create_instance_group_m2m... OK", "  Applying main.0074_v360_migrate_instance_group_relations... OK", "  Applying main.0075_v360_remove_old_instance_group_relations... OK", "  Applying main.0076_v360_add_new_instance_group_relations... OK", "  Applying main.0077_v360_add_default_orderings... OK", "  Applying main.0078_v360_clear_sessions_tokens_jt... OK", "  Applying main.0079_v360_rm_implicit_oauth2_apps... OK", "  Applying main.0080_v360_replace_job_origin... OK", "  Applying main.0081_v360_notify_on_start... OK", "  Applying main.0082_v360_webhook_http_method... OK", "  Applying main.0083_v360_job_branch_override... OK", "  Applying main.0084_v360_token_description... OK", "  Applying main.0085_v360_add_notificationtemplate_messages... OK", "  Applying main.0086_v360_workflow_approval... OK", "  Applying main.0087_v360_update_credential_injector_help_text... OK", "  Applying main.0088_v360_dashboard_optimizations... OK", "  Applying main.0089_v360_new_job_event_types... OK", "  Applying main.0090_v360_WFJT_prompts... OK", "  Applying main.0091_v360_approval_node_notifications... OK", "  Applying main.0092_v360_webhook_mixin... OK", "  Applying main.0093_v360_personal_access_tokens... OK", "  Applying main.0094_v360_webhook_mixin2... OK", "  Applying main.0095_v360_increase_instance_version_length... OK", "  Applying main.0096_v360_container_groups... OK", "  Applying main.0097_v360_workflowapproval_approved_or_denied_by... OK", "  Applying main.0098_v360_rename_cyberark_aim_credential_type... OK", "  Applying main.0099_v361_license_cleanup... OK", "  Applying main.0100_v370_projectupdate_job_tags... OK", "  Applying main.0101_v370_generate_new_uuids_for_iso_nodes... OK", "  Applying main.0102_v370_unifiedjob_canceled... OK", "  Applying main.0103_v370_remove_computed_fields... OK", "  Applying main.0104_v370_cleanup_old_scan_jts... OK", "  Applying main.0105_v370_remove_jobevent_parent_and_hosts... OK", "  Applying main.0106_v370_remove_inventory_groups_with_active_failures... OK", "  Applying main.0107_v370_workflow_convergence_api_toggle... OK", "  Applying main.0108_v370_unifiedjob_dependencies_processed... OK", "  Applying oauth2_provider.0002_08_updates... OK", "  Applying oauth2_provider.0003_auto_20160316_1503... OK", "  Applying oauth2_provider.0004_auto_20160525_1623... OK", "  Applying oauth2_provider.0005_auto_20170514_1141... OK", "  Applying oauth2_provider.0006_auto_20171214_2232... OK", "  Applying sites.0001_initial... OK", "  Applying sites.0002_alter_domain_unique... OK", "  Applying social_django.0001_initial... OK", "  Applying social_django.0002_add_related_name... OK", "  Applying social_django.0003_alter_email_max_length... OK", "  Applying social_django.0004_auto_20160423_0400... OK", "  Applying social_django.0005_auto_20160727_2333... OK", "  Applying social_django.0006_partial... OK", "  Applying social_django.0007_code_timestamp... OK", "  Applying social_django.0008_partial_timestamp... OK", "  Applying sso.0001_initial... OK", "  Applying sso.0002_expand_provider_options... OK", "  Applying taggit.0003_taggeditem_add_unique_index... OK"]}

TASK [tower : include_tasks] ***************************************************
task path: /opt/ansible/roles/tower/tasks/main.yml:54
[WARNING]: While constructing a mapping from
/opt/ansible/roles/tower/tasks/initialize.yml, line 2, column 3, found a
duplicate dict key (changed_when). Using last defined value only.
included: /opt/ansible/roles/tower/tasks/initialize.yml for localhost

TASK [tower : Check if there are any Tower super users defined.] ***************
task path: /opt/ansible/roles/tower/tasks/initialize.yml:2
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "kubectl exec -n awx awx-tower-web-5b7bc9f756-8zlhh -- bash -c \"echo 'from django.contrib.auth.models import User; nsu = User.objects.filter(is_superuser=True).count(); exit(0 if nsu > 0 else 1)' | awx-manage shell\"", "delta": "0:00:04.617848", "end": "2020-03-16 10:22:07.705351", "msg": "non-zero return code", "rc": 1, "start": "2020-03-16 10:22:03.087503", "stderr": "command terminated with exit code 1", "stderr_lines": ["command terminated with exit code 1"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [tower : Create Tower super user via Django if it doesn't exist.] *********
task path: /opt/ansible/roles/tower/tasks/initialize.yml:14
changed: [localhost] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}

TASK [tower : Create preload data if necessary.] *******************************
task path: /opt/ansible/roles/tower/tasks/initialize.yml:23
changed: [localhost] => {"changed": true, "cmd": "kubectl exec -n awx awx-tower-web-5b7bc9f756-8zlhh -- bash -c \"awx-manage create_preload_data\"", "delta": "0:00:05.741234", "end": "2020-03-16 10:22:20.515205", "rc": 0, "start": "2020-03-16 10:22:14.773971", "stderr": "", "stderr_lines": [], "stdout": "Default organization added.\nDemo Credential, Inventory, and Job Template added.\n(changed: True)", "stdout_lines": ["Default organization added.", "Demo Credential, Inventory, and Job Template added.", "(changed: True)"]}

TASK [tower : Scale the tower_task deployment to 0 replicas after migration.] ***
task path: /opt/ansible/roles/tower/tasks/main.yml:56
changed: [localhost] => {"changed": true, "method": "patch", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"annotations": {"deployment.kubernetes.io/revision": "1"}, "creationTimestamp": "2020-03-16T10:18:31Z", "generation": 2, "labels": {"app": "tower-task"}, "name": "awx-tower-task", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958747", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-tower-task", "uid": "92ec8294-39a8-4e85-b6c8-fcb02c0ec696"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 0, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-task"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-task"}}, "spec": {"containers": [{"command": ["/usr/bin/launch_awx_task.sh"], "envFrom": [{"configMapRef": {"name": "awx-tower-configmap"}}, {"secretRef": {"name": "awx-tower-secret"}}], "image": "ansible/awx_task:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower-task", "resources": {"requests": {"cpu": "500m", "memory": "1Gi"}}, "securityContext": {"privileged": true}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}]}}}, "status": {"availableReplicas": 1, "conditions": [{"lastTransitionTime": "2020-03-16T10:18:39Z", "lastUpdateTime": "2020-03-16T10:18:39Z", "message": "Deployment has minimum availability.", "reason": "MinimumReplicasAvailable", "status": "True", "type": "Available"}, {"lastTransitionTime": "2020-03-16T10:18:31Z", "lastUpdateTime": "2020-03-16T10:18:39Z", "message": "ReplicaSet \"awx-tower-task-56777cb4c8\" has successfully progressed.", "reason": "NewReplicaSetAvailable", "status": "True", "type": "Progressing"}], "observedGeneration": 1, "readyReplicas": 1, "replicas": 1, "updatedReplicas": 1}}}

TASK [tower : Scale the tower_task deployment back to 1 replica after migration.] ***
task path: /opt/ansible/roles/tower/tasks/main.yml:63
changed: [localhost] => {"changed": true, "method": "patch", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"annotations": {"deployment.kubernetes.io/revision": "1"}, "creationTimestamp": "2020-03-16T10:18:31Z", "generation": 3, "labels": {"app": "tower-task"}, "name": "awx-tower-task", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958762", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-tower-task", "uid": "92ec8294-39a8-4e85-b6c8-fcb02c0ec696"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-task"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-task"}}, "spec": {"containers": [{"command": ["/usr/bin/launch_awx_task.sh"], "envFrom": [{"configMapRef": {"name": "awx-tower-configmap"}}, {"secretRef": {"name": "awx-tower-secret"}}], "image": "ansible/awx_task:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower-task", "resources": {"requests": {"cpu": "500m", "memory": "1Gi"}}, "securityContext": {"privileged": true}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}]}}}, "status": {"conditions": [{"lastTransitionTime": "2020-03-16T10:18:39Z", "lastUpdateTime": "2020-03-16T10:18:39Z", "message": "Deployment has minimum availability.", "reason": "MinimumReplicasAvailable", "status": "True", "type": "Available"}, {"lastTransitionTime": "2020-03-16T10:18:31Z", "lastUpdateTime": "2020-03-16T10:18:39Z", "message": "ReplicaSet \"awx-tower-task-56777cb4c8\" has successfully progressed.", "reason": "NewReplicaSetAvailable", "status": "True", "type": "Progressing"}], "observedGeneration": 2}}}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************
localhost                  : ok=11   changed=7    unreachable=0    failed=0    skipped=1    rescued=0    ignored=1   

/tmp/ansible-operator/runner/tower.ansible.com/v1alpha1/Tower/awx/awx/artifacts/4751997750760398084//stdout
ansible-playbook 2.9.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: main.yml *************************************************************
1 plays in /opt/ansible/main.yml

PLAY [localhost] ***************************************************************
META: ran handlers

TASK [tower : Ensure configured Tower resources exist in the cluster.] *********
task path: /opt/ansible/roles/tower/tasks/main.yml:2
ok: [localhost] => (item=tower_memcached.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "item": "tower_memcached.yaml.j2", "result": {"results": [{"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"annotations": {"deployment.kubernetes.io/revision": "1"}, "creationTimestamp": "2020-03-16T10:18:24Z", "generation": 1, "labels": {"app": "tower-memcached"}, "name": "awx-memcached", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958092", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-memcached", "uid": "354b9c58-3453-4a5b-ba49-2802341eb4fe"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-memcached"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-memcached"}}, "spec": {"containers": [{"image": "memcached:alpine", "imagePullPolicy": "IfNotPresent", "name": "memcached", "ports": [{"containerPort": 11211, "protocol": "TCP"}], "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File"}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30}}}, "status": {"availableReplicas": 1, "conditions": [{"lastTransitionTime": "2020-03-16T10:18:25Z", "lastUpdateTime": "2020-03-16T10:18:25Z", "message": "Deployment has minimum availability.", "reason": "MinimumReplicasAvailable", "status": "True", "type": "Available"}, {"lastTransitionTime": "2020-03-16T10:18:24Z", "lastUpdateTime": "2020-03-16T10:18:25Z", "message": "ReplicaSet \"awx-memcached-587b55d5fd\" has successfully progressed.", "reason": "NewReplicaSetAvailable", "status": "True", "type": "Progressing"}], "observedGeneration": 1, "readyReplicas": 1, "replicas": 1, "updatedReplicas": 1}}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:24Z", "labels": {"app": "tower-memcached"}, "name": "awx-memcached", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958079", "selfLink": "/api/v1/namespaces/awx/services/awx-memcached", "uid": "10eafa1e-6c1f-4441-9f75-fa5a9d42540d"}, "spec": {"clusterIP": "None", "ports": [{"port": 11211, "protocol": "TCP", "targetPort": 11211}], "selector": {"app": "tower-memcached"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}]}}
ok: [localhost] => (item=tower_postgres.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "item": "tower_postgres.yaml.j2", "result": {"results": [{"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "v1", "data": {"password": "YXd4cGFzcw=="}, "kind": "Secret", "metadata": {"creationTimestamp": "2020-03-16T10:18:25Z", "name": "awx-postgres-pass", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958087", "selfLink": "/api/v1/namespaces/awx/secrets/awx-postgres-pass", "uid": "59a0c17a-4829-40ae-972f-f66cc86ae4f8"}, "type": "Opaque"}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "apps/v1", "kind": "StatefulSet", "metadata": {"creationTimestamp": "2020-03-16T10:18:26Z", "generation": 1, "labels": {"app": "tower-postgres"}, "name": "awx-postgres", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958161", "selfLink": "/apis/apps/v1/namespaces/awx/statefulsets/awx-postgres", "uid": "3d64a660-2bfb-4b52-9473-db66446509c3"}, "spec": {"podManagementPolicy": "OrderedReady", "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-postgres"}}, "serviceName": "awx", "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-postgres"}}, "spec": {"containers": [{"env": [{"name": "POSTGRES_DB", "value": "awx"}, {"name": "POSTGRES_USER", "value": "awx"}, {"name": "POSTGRES_PASSWORD", "valueFrom": {"secretKeyRef": {"key": "password", "name": "awx-postgres-pass"}}}], "image": "postgres:10", "imagePullPolicy": "IfNotPresent", "name": "postgres", "ports": [{"containerPort": 3306, "name": "postgres", "protocol": "TCP"}], "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/var/lib/postgresql/data", "name": "postgres", "subPath": "data"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30}}, "updateStrategy": {"type": "RollingUpdate"}, "volumeClaimTemplates": [{"apiVersion": "v1", "kind": "PersistentVolumeClaim", "metadata": {"creationTimestamp": null, "name": "postgres"}, "spec": {"accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "8Gi"}}, "volumeMode": "Filesystem"}, "status": {"phase": "Pending"}}]}, "status": {"collisionCount": 0, "currentReplicas": 1, "currentRevision": "awx-postgres-5599b677", "observedGeneration": 1, "readyReplicas": 1, "replicas": 1, "updateRevision": "awx-postgres-5599b677", "updatedReplicas": 1}}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:26Z", "labels": {"app": "tower-postgres"}, "name": "awx-postgres", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958096", "selfLink": "/api/v1/namespaces/awx/services/awx-postgres", "uid": "4ceb15a7-fee6-4ebb-aa36-82c832661c67"}, "spec": {"clusterIP": "None", "ports": [{"port": 5432, "protocol": "TCP", "targetPort": 5432}], "selector": {"app": "tower-postgres"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}]}}
ok: [localhost] => (item=tower_rabbitmq.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "item": "tower_rabbitmq.yaml.j2", "result": {"results": [{"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"annotations": {"deployment.kubernetes.io/revision": "1"}, "creationTimestamp": "2020-03-16T10:18:27Z", "generation": 1, "labels": {"app": "tower-rabbitmq"}, "name": "awx-rabbitmq", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958153", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-rabbitmq", "uid": "1c3e3bc9-f8e3-4ddb-ab69-02b5fe842dce"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-rabbitmq"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-rabbitmq"}}, "spec": {"containers": [{"env": [{"name": "RABBITMQ_DEFAULT_VHOST", "value": "awx"}, {"name": "RABBITMQ_NODE_PORT", "value": "5672"}], "image": "rabbitmq:3", "imagePullPolicy": "IfNotPresent", "name": "rabbitmq", "ports": [{"containerPort": 15672, "protocol": "TCP"}, {"containerPort": 5672, "protocol": "TCP"}], "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File"}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30}}}, "status": {"availableReplicas": 1, "conditions": [{"lastTransitionTime": "2020-03-16T10:18:29Z", "lastUpdateTime": "2020-03-16T10:18:29Z", "message": "Deployment has minimum availability.", "reason": "MinimumReplicasAvailable", "status": "True", "type": "Available"}, {"lastTransitionTime": "2020-03-16T10:18:27Z", "lastUpdateTime": "2020-03-16T10:18:29Z", "message": "ReplicaSet \"awx-rabbitmq-7f8f6ff647\" has successfully progressed.", "reason": "NewReplicaSetAvailable", "status": "True", "type": "Progressing"}], "observedGeneration": 1, "readyReplicas": 1, "replicas": 1, "updatedReplicas": 1}}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:27Z", "labels": {"app": "tower-rabbitmq"}, "name": "awx-rabbitmq", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958125", "selfLink": "/api/v1/namespaces/awx/services/awx-rabbitmq", "uid": "b811d4df-9277-485c-b946-588a8e60a8d5"}, "spec": {"clusterIP": "None", "ports": [{"port": 5672, "protocol": "TCP", "targetPort": 5672}], "selector": {"app": "tower-rabbitmq"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}]}}
fault-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/\";\n\n        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)\n        add_header X-Frame-Options \"DENY\";\n\n        location /nginx_status {\n            stub_status on;\n            access_log off;\n            allow 127.0.0.1;\n            deny all;\n        }\n\n        location /static/ {\n            alias /var/lib/awx/public/static/;\n        }\n\n        location /favicon.ico {\n            alias /var/lib/awx/public/static/favicon.ico;\n        }\n\n        location /websocket {\n            # Pass request to the upstream alias\n            proxy_pass http://daphne;\n            # Require http version 1.1 to allow for upgrade requests\n            proxy_http_version 1.1;\n            # We want proxy_buffering off for proxying to websockets.\n            proxy_buffering off;\n            # http://en.wikipedia.org/wiki/X-Forwarded-For\n            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n            # enable this if you use HTTPS:\n            proxy_set_header X-Forwarded-Proto https;\n            # pass the Host: header from the client for the sake of redirects\n            proxy_set_header Host $http_host;\n            # We've set the Host header, so we don't need Nginx to muddle\n            # about with redirects\n            proxy_redirect off;\n            # Depending on the request value, set the Upgrade and\n            # connection headers\n            proxy_set_header Upgrade $http_upgrade;\n            proxy_set_header Connection $connection_upgrade;\n        }\n\n        location / {\n            # Add trailing / if missing\n            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;\n            uwsgi_read_timeout 120s;\n            uwsgi_pass uwsgi;\n            include /etc/nginx/uwsgi_params;                proxy_set_header X-Forwarded-Port 443;\n        }\n    }\n}\n", "settings": "import os\nimport socket\n\ndef get_secret():\n    if os.path.exists(\"/etc/tower/SECRET_KEY\"):\n        return open('/etc/tower/SECRET_KEY', 'rb').read().strip()\n\nADMINS = ()\nSTATIC_ROOT = '/var/lib/awx/public/static'\nPROJECTS_ROOT = '/var/lib/awx/projects'\nJOBOUTPUT_ROOT = '/var/lib/awx/job_status'\n\nSECRET_KEY = get_secret()\n\nALLOWED_HOSTS = ['*']\n\nINTERNAL_API_URL = 'http://127.0.0.1:8052'\n\n# Container environments don't like chroots\nAWX_PROOT_ENABLED = False\n\n# Automatically deprovision pods that go offline\nAWX_AUTO_DEPROVISION_INSTANCES = True\n\nCLUSTER_HOST_ID = socket.gethostname()\nSYSTEM_UUID = '00000000-0000-0000-0000-000000000000'\n\nCSRF_COOKIE_SECURE = False\nSESSION_COOKIE_SECURE = False\n\nSERVER_EMAIL = 'root@localhost'\nDEFAULT_FROM_EMAIL = 'webmaster@localhost'\nEMAIL_SUBJECT_PREFIX = '[AWX] '\n\nEMAIL_HOST = 'localhost'\nEMAIL_PORT = 25\nEMAIL_HOST_USER = ''\nEMAIL_HOST_PASSWORD = ''\nEMAIL_USE_TLS = False\n\nLOGGING['handlers']['console'] = {\n    '()': 'logging.StreamHandler',\n    'level': 'DEBUG',\n    'formatter': 'simple',\n}\n\nLOGGING['loggers']['django.request']['handlers'] = ['console']\nLOGGING['loggers']['rest_framework.request']['handlers'] = ['console']\nLOGGING['loggers']['awx']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.commands.run_callback_receiver']['handlers'] = ['console']\nLOGGING['loggers']['awx.main.tasks']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.scheduler']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['django_auth_ldap']['handlers'] = ['console']\nLOGGING['loggers']['social']['handlers'] = ['console']\nLOGGING['loggers']['system_tracking_migrations']['handlers'] = ['console']\nLOGGING['loggers']['rbac_migrations']['handlers'] = ['console']\nLOGGING['loggers']['awx.isolated.manager.playbooks']['handlers'] = ['console']\nLOGGING['handlers']['callback_receiver'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['task_system'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['tower_warnings'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['rbac_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['system_tracking_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['management_playbooks'] = {'class': 'logging.NullHandler'}\n\nDATABASES = {\n    'default': {\n        'ATOMIC_REQUESTS': True,\n        'ENGINE': 'awx.main.db.profiled_pg',\n        'NAME': 'awx',\n        'USER': 'awx',\n        'PASSWORD': 'awxpass',\n        'HOST': 'awx-postgres.awx.svc.cluster.local',\n        'PORT': '5432',\n    }\n}\n\nif os.getenv(\"DATABASE_SSLMODE\", False):\n    DATABASES['default']['OPTIONS'] = {'sslmode': os.getenv(\"DATABASE_SSLMODE\")}\n\nCACHES = {\n    'default': {\n        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',\n        'LOCATION': '{}:{}'.format(\"awx-memcached.awx.svc.cluster.local\", \"11211\")\n    },\n    'ephemeral': {\n        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',\n    },\n}\n\nBROKER_URL = 'amqp://{}:{}@{}:{}/{}'.format(\n    'guest',\n    'guest',\n    'awx-rabbitmq.awx.svc.cluster.local',\n    '5672',\n    'awx')\n\nCHANNEL_LAYERS = {\n    'default': {'BACKEND': 'asgi_amqp.AMQPChannelLayer',\n                'ROUTING': 'awx.main.routing.channel_routing',\n                'CONFIG': {'url': BROKER_URL}}\n}\n\nUSE_X_FORWARDED_PORT = True\n"}, "kind": "ConfigMap", "metadata": {"creationTimestamp": "2020-03-16T10:18:28Z", "labels": {"app": "tower"}, "name": "awx-tower-configmap", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958144", "selfLink": "/api/v1/namespaces/awx/configmaps/awx-tower-configmap", "uid": "399ccb43-8c32-481d-aok: [localhost] => (item=tower_config.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "item": "tower_config.yaml.j2", "method": "patch", "result": {"apiVersion": "v1", "data": {"environment": "DATABASE_USER=awx\nDATABASE_NAME=awx\nDATABASE_HOST='awx-postgres.awx.svc.cluster.local'\nDATABASE_PORT='5432'\nDATABASE_PASSWORD=awxpass\nMEMCACHED_HOST='awx-memcached.awx.svc.cluster.local'\nMEMCACHED_PORT='11211'\nRABBITMQ_HOST='awx-rabbitmq.awx.svc.cluster.local'\nRABBITMQ_PORT='5672'\nAWX_SKIP_MIGRATIONS=true\n", "nginx_conf": "worker_processes  1;\npid        /tmp/nginx.pid;\n\nevents {\n    worker_connections  1024;\n}\n\nhttp {\n    include       /etc/nginx/mime.types;\n    default_type  application/octet-stream;\n    server_tokens off;\n\n    log_format  main  '$remote_addr - $remote_user [$time_local] \"$request\" '\n                      '$status $body_bytes_sent \"$http_referer\" '\n                      '\"$http_user_agent\" \"$http_x_forwarded_for\"';\n\n    access_log /dev/stdout main;\n\n    map $http_upgrade $connection_upgrade {\n        default upgrade;\n        ''      close;\n    }\n\n    sendfile        on;\n    #tcp_nopush     on;\n    #gzip  on;\n\n    upstream uwsgi {\n        server 127.0.0.1:8050;\n    }\n\n    upstream daphne {\n        server 127.0.0.1:8051;\n    }\n\n    server {\n        listen 8052 default_server;\n\n        # If you have a domain name, this is where to add it\n        server_name _;\n        keepalive_timeout 65;\n\n        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)\n        add_header Strict-Transport-Security max-age=15768000;\n        add_header Content-Security-Policy \"default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/\";\n        add_header X-Content-Security-Policy \"default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/\";\n\n        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)\n        add_header X-Frame-Options \"DENY\";\n\n        location /nginx_status {\n            stub_status on;\n            access_log off;\n            allow 127.0.0.1;\n            deny all;\n        }\n\n        location /static/ {\n            alias /var/lib/awx/public/static/;\n        }\n\n        location /favicon.ico {\n            alias /var/lib/awx/public/static/favicon.ico;\n        }\n\n        location /websocket {\n            # Pass request to the upstream alias\n            proxy_pass http://daphne;\n            # Require http version 1.1 to allow for upgrade requests\n            proxy_http_version 1.1;\n            # We want proxy_buffering off for proxying to websockets.\n            proxy_buffering off;\n            # http://en.wikipedia.org/wiki/X-Forwarded-For\n            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n            # enable this if you use HTTPS:\n            proxy_set_header X-Forwarded-Proto https;\n            # pass the Host: header from the client for the sake of redirects\n            proxy_set_header Host $http_host;\n            # We've set the Host header, so we don't need Nginx to muddle\n            # about with redirects\n            proxy_redirect off;\n            # Depending on the request value, set the Upgrade and\n            # connection headers\n            proxy_set_header Upgrade $http_upgrade;\n            proxy_set_header Connection $connection_upgrade;\n        }\n\n        location / {\n            # Add trailing / if missing\n            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;\n            uwsgi_read_timeout 120s;\n            uwsgi_pass uwsgi;\n            include /etc/nginx/uwsgi_params;                proxy_set_header X-Forwarded-Port 443;\n        }\n    }\n}\n", "settings": "import os\nimport socket\n\ndef get_secret():\n    if os.path.exists(\"/etc/tower/SECRET_KEY\"):\n        return open('/etc/tower/SECRET_KEY', 'rb').read().strip()\n\nADMINS = ()\nSTATIC_ROOT = '/var/lib/awx/public/static'\nPROJECTS_ROOT = '/var/lib/awx/projects'\nJOBOUTPUT_ROOT = '/var/lib/awx/job_status'\n\nSECRET_KEY = get_secret()\n\nALLOWED_HOSTS = ['*']\n\nINTERNAL_API_URL = 'http://127.0.0.1:8052'\n\n# Container environments don't like chroots\nAWX_PROOT_ENABLED = False\n\n# Automatically deprovision pods that go offline\nAWX_AUTO_DEPROVISION_INSTANCES = True\n\nCLUSTER_HOST_ID = socket.gethostname()\nSYSTEM_UUID = '00000000-0000-0000-0000-000000000000'\n\nCSRF_COOKIE_SECURE = False\nSESSION_COOKIE_SECURE = False\n\nSERVER_EMAIL = 'root@localhost'\nDEFAULT_FROM_EMAIL = 'webmaster@localhost'\nEMAIL_SUBJECT_PREFIX = '[AWX] '\n\nEMAIL_HOST = 'localhost'\nEMAIL_PORT = 25\nEMAIL_HOST_USER = ''\nEMAIL_HOST_PASSWORD = ''\nEMAIL_USE_TLS = False\n\nLOGGING['handlers']['console'] = {\n    '()': 'logging.StreamHandler',\n    'level': 'DEBUG',\n    'formatter': 'simple',\n}\n\nLOGGING['loggers']['django.request']['handlers'] = ['console']\nLOGGING['loggers']['rest_framework.request']['handlers'] = ['console']\nLOGGING['loggers']['awx']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.commands.run_callback_receiver']['handlers'] = ['console']\nLOGGING['loggers']['awx.main.tasks']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['awx.main.scheduler']['handlers'] = ['console', 'external_logger']\nLOGGING['loggers']['django_auth_ldap']['handlers'] = ['console']\nLOGGING['loggers']['social']['handlers'] = ['console']\nLOGGING['loggers']['system_tracking_migrations']['handlers'] = ['console']\nLOGGING['loggers']['rbac_migrations']['handlers'] = ['console']\nLOGGING['loggers']['awx.isolated.manager.playbooks']['handlers'] = ['console']\nLOGGING['handlers']['callback_receiver'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['task_system'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['tower_warnings'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['rbac_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['system_tracking_migrations'] = {'class': 'logging.NullHandler'}\nLOGGING['handlers']['management_playbooks'] = {'class': 'logging.NullHandler'}\n\nDATABASES = {\n    'default': {\n        'ATOMIC_REQUESTS': True,\n        'ENGINE': 'awx.main.db.profiled_pg',\n        'NAME': 'awx',\n        'USER': 'awx',\n        'PASSWORD': 'awxpass',\n        'HOST': 'awx-postgres.awx.svc.cluster.local',\n        'PORT': '5432',\n    }\n}\n\nif os.getenv(\"DATABASE_SSLMODE\", False):\n    DATABASES['default']['OPTIONS'] = {'sslmode': os.getenv(\"DATABASE_SSLMODE\")}\n\nCACHES = {\n    'default': {\n        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',\n        'LOCATION': '{}:{}'.format(\"awx-memcached.awx.svc.cluster.local\", \"11211\")\n    },\n    'ephemeral': {\n        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',\n    },\n}\n\nBROKER_URL = 'amqp://{}:{}@{}:{}/{}'.format(\n    'guest',\n    'guest',\n    'awx-rabbitmq.awx.svc.cluster.local',\n    '5672',\n    'awx')\n\nCHANNEL_LAYERS = {\n    'default': {'BACKEND': 'asgi_amqp.AMQPChannelLayer',\n                'ROUTING': 'awx.main.routing.channel_routing',\n                'CONFIG': {'url': BROKER_URL}}\n}\n\nUSE_X_FORWARDED_PORT = True\n"}, "kind": "ConfigMap", "metadata": {"creationTimestamp": "2020-03-16T10:18:28Z", "labels": {"app": "tower"}, "name": "awx-tower-configmap", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958144", "selfLink": "/api/v1/namespaces/awx/configmaps/awx-tower-configmap", "uid": "399ccb43-8c32-481d-aba3-617c3395d8cf"}}}
/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}, {"mountPath": "/etc/nginx/nginx.conf", "name": "nginx-conf", "readOnly": true, "subPath": "nginx.conf"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}, {"configMap": {"defaultMode": 420, "items": [{"key": "nginx_conf", "path": "nginx.conf"}], "name": "awx-tower-configmap"}, "name": "nginx-conf"}]}}}, "status": {"availableReplicas": 1, "conditions": [{"lastTransitionTime": "2020-03-16T10:18:32Z", "lastUpdateTime": "2020-03-16T10:18:32Z", "message": "Deployment has minimum availability.", "reason": "MinimumReplicasAvailable", "status": "True", "type": "Available"}, {"lastTransitionTime": "2020-03-16T10:18:30Z", "lastUpdateTime": "2020-03-16T10:18:32Z", "message": "ReplicaSet \"awx-tower-web-5b7bc9f756\" has successfully progressed.", "reason": "NewReplicaSetAvailable", "status": "True", "type": "Progressing"}], "observedGeneration": 1, "readyReplicas": 1, "replicas": 1, "updatedReplicas": 1}}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "v1", "kind": "Service", "metadata": ok: [localhost] => (item=tower_web.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "item": "tower_web.yaml.j2", "result": {"results": [{"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "v1", "data": {"admin_password": "cHdBbjgyIXNp", "secret_key": "YXd4c2VjcmV0"}, "kind": "Secret", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "name": "awx-tower-secret", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958158", "selfLink": "/api/v1/namespaces/awx/secrets/awx-tower-secret", "uid": "8280226f-724f-40b3-b31f-d30ef427e107"}, "type": "Opaque"}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"annotations": {"deployment.kubernetes.io/revision": "1"}, "creationTimestamp": "2020-03-16T10:18:30Z", "generation": 1, "labels": {"app": "tower"}, "name": "awx-tower-web", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958203", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-tower-web", "uid": "645f9429-d68b-4a7c-9f03-fe8a268197da"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower"}}, "spec": {"containers": [{"image": "ansible/awx_web:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower", "ports": [{"containerPort": 8052, "protocol": "TCP"}], "resources": {"requests": {"cpu": "1", "memory": "2Gi"}}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}, {"mountPath": "/etc/nginx/nginx.conf", "name": "nginx-conf", "readOnly": true, "subPath": "nginx.conf"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}, {"configMap": {"defaultMode": 420, "items": [{"key": "nginx_conf", "path": "nginx.conf"}], "name": "awx-tower-configmap"}, "name": "nginx-conf"}]}}}, "status": {"availableReplicas": 1, "conditions": [{"lastTransitionTime": "2020-03-16T10:18:32Z", "lastUpdateTime": "2020-03-16T10:18:32Z", "message": "Deployment has minimum availability.", "reason": "MinimumReplicasAvailable", "status": "True", "type": "Available"}, {"lastTransitionTime": "2020-03-16T10:18:30Z", "lastUpdateTime": "2020-03-16T10:18:32Z", "message": "ReplicaSet \"awx-tower-web-5b7bc9f756\" has successfully progressed.", "reason": "NewReplicaSetAvailable", "status": "True", "type": "Progressing"}], "observedGeneration": 1, "readyReplicas": 1, "replicas": 1, "updatedReplicas": 1}}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "v1", "kind": "Service", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "labels": {"app": "tower"}, "name": "awx-service", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958173", "selfLink": "/api/v1/namespaces/awx/services/awx-service", "uid": "9c2c6278-f50b-4513-bbba-b83ff1abdd6e"}, "spec": {"clusterIP": "10.152.183.136", "ports": [{"port": 80, "protocol": "TCP", "targetPort": 8052}], "selector": {"app": "tower"}, "sessionAffinity": "None", "type": "ClusterIP"}, "status": {"loadBalancer": {}}}, "warnings": []}, {"changed": false, "diff": {}, "method": "patch", "result": {"apiVersion": "extensions/v1beta1", "kind": "Ingress", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "generation": 1, "name": "awx-ingress", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958301", "selfLink": "/apis/extensions/v1beta1/namespaces/awx/ingresses/awx-ingress", "uid": "800855e4-1c8e-4aac-9f0e-61a4a7ec6d50"}, "spec": {"rules": [{"host": "minikube", "http": {"paths": [{"backend": {"serviceName": "awx-service", "servicePort": 80}, "path": "/"}]}}]}, "status": {"loadBalancer": {"ingress": [{"ip": "127.0.0.1"}]}}}, "warnings": []}]}}
ok: [localhost] => (item=tower_task.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "item": "tower_task.yaml.j2", "method": "patch", "result": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"annotations": {"deployment.kubernetes.io/revision": "1"}, "creationTimestamp": "2020-03-16T10:18:31Z", "generation": 3, "labels": {"app": "tower-task"}, "name": "awx-tower-task", "namespace": "awx", "ownerReferences": [{"apiVersion": "tower.ansible.com/v1alpha1", "kind": "Tower", "name": "awx", "uid": "26077c32-311b-4897-8cd8-1b75303a1a29"}], "resourceVersion": "958789", "selfLink": "/apis/apps/v1/namespaces/awx/deployments/awx-tower-task", "uid": "92ec8294-39a8-4e85-b6c8-fcb02c0ec696"}, "spec": {"progressDeadlineSeconds": 600, "replicas": 1, "revisionHistoryLimit": 10, "selector": {"matchLabels": {"app": "tower-task"}}, "strategy": {"rollingUpdate": {"maxSurge": "25%", "maxUnavailable": "25%"}, "type": "RollingUpdate"}, "template": {"metadata": {"creationTimestamp": null, "labels": {"app": "tower-task"}}, "spec": {"containers": [{"command": ["/usr/bin/launch_awx_task.sh"], "envFrom": [{"configMapRef": {"name": "awx-tower-configmap"}}, {"secretRef": {"name": "awx-tower-secret"}}], "image": "ansible/awx_task:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower-task", "resources": {"requests": {"cpu": "500m", "memory": "1Gi"}}, "securityContext": {"privileged": true}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}]}}}, "status": {"availableReplicas": 1, "conditions": [{"lastTransitionTime": "2020-03-16T10:18:31Z", "lastUpdateTime": "2020-03-16T10:18:39Z", "message": "ReplicaSet \"awx-tower-task-56777cb4c8\" has successfully progressed.", "reason": "NewReplicaSetAvailable", "status": "True", "type": "Progressing"}, {"lastTransitionTime": "2020-03-16T10:22:27Z", "lastUpdateTime": "2020-03-16T10:22:27Z", "message": "Deployment has minimum availability.", "reason": "MinimumReplicasAvailable", "status": "True", "type": "Available"}], "observedGeneration": 3, "readyReplicas": 1, "replicas": 1, "updatedReplicas": 1}}}

TASK [tower : Get the Tower pod information.] **********************************
task path: /opt/ansible/roles/tower/tasks/main.yml:14
[DEPRECATION WARNING]: The 'k8s_facts' module has been renamed to 'k8s_info'. 
This feature will be removed in version 2.13. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost] => {"attempts": 1, "changed": false, "resources": [{"apiVersion": "v1", "kind": "Pod", "metadata": {"creationTimestamp": "2020-03-16T10:18:30Z", "generateName": "awx-tower-web-5b7bc9f756-", "labels": {"app": "tower", "pod-template-hash": "5b7bc9f756"}, "name": "awx-tower-web-5b7bc9f756-8zlhh", "namespace": "awx", "ownerReferences": [{"apiVersion": "apps/v1", "blockOwnerDeletion": true, "controller": true, "kind": "ReplicaSet", "name": "awx-tower-web-5b7bc9f756", "uid": "9d7b8bb7-ea82-48aa-8e40-6658079079aa"}], "resourceVersion": "958200", "selfLink": "/api/v1/namespaces/awx/pods/awx-tower-web-5b7bc9f756-8zlhh", "uid": "820269a4-1534-4782-9a71-2e60a6b8ff43"}, "spec": {"containers": [{"image": "ansible/awx_web:9.2.0", "imagePullPolicy": "IfNotPresent", "name": "tower", "ports": [{"containerPort": 8052, "protocol": "TCP"}], "resources": {"requests": {"cpu": "1", "memory": "2Gi"}}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/etc/tower/SECRET_KEY", "name": "secret-key", "readOnly": true, "subPath": "SECRET_KEY"}, {"mountPath": "/etc/tower/conf.d/environment.sh", "name": "environment", "readOnly": true, "subPath": "environment.sh"}, {"mountPath": "/etc/tower/settings.py", "name": "settings", "readOnly": true, "subPath": "settings.py"}, {"mountPath": "/etc/nginx/nginx.conf", "name": "nginx-conf", "readOnly": true, "subPath": "nginx.conf"}, {"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount", "name": "default-token-sd5hr", "readOnly": true}]}], "dnsPolicy": "ClusterFirst", "enableServiceLinks": true, "nodeName": "minikube", "priority": 0, "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "default", "serviceAccountName": "default", "terminationGracePeriodSeconds": 30, "tolerations": [{"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists", "tolerationSeconds": 300}, {"effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists", "tolerationSeconds": 300}], "volumes": [{"name": "secret-key", "secret": {"defaultMode": 420, "items": [{"key": "secret_key", "path": "SECRET_KEY"}], "secretName": "awx-tower-secret"}}, {"configMap": {"defaultMode": 420, "items": [{"key": "environment", "path": "environment.sh"}], "name": "awx-tower-configmap"}, "name": "environment"}, {"configMap": {"defaultMode": 420, "items": [{"key": "settings", "path": "settings.py"}], "name": "awx-tower-configmap"}, "name": "settings"}, {"configMap": {"defaultMode": 420, "items": [{"key": "nginx_conf", "path": "nginx.conf"}], "name": "awx-tower-configmap"}, "name": "nginx-conf"}, {"name": "default-token-sd5hr", "secret": {"defaultMode": 420, "secretName": "default-token-sd5hr"}}]}, "status": {"conditions": [{"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:30Z", "status": "True", "type": "Initialized"}, {"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:32Z", "status": "True", "type": "Ready"}, {"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:32Z", "status": "True", "type": "ContainersReady"}, {"lastProbeTime": null, "lastTransitionTime": "2020-03-16T10:18:30Z", "status": "True", "type": "PodScheduled"}], "containerStatuses": [{"containerID": "containerd://026f570dcda574d82952333914bcf189578830892cbe472fc1829b230cb597ad", "image": "docker.io/ansible/awx_web:9.2.0", "imageID": "docker.io/ansible/awx_web@sha256:57232e6820eb1bfad3c6910bd993e1a8c0d644c0c0fcfb75a25a0b51b3d6fdec", "lastState": {}, "name": "tower", "ready": true, "restartCount": 0, "started": true, "state": {"running": {"startedAt": "2020-03-16T10:18:32Z"}}}], "hostIP": "192.168.0.121", "phase": "Running", "podIP": "10.1.90.166", "podIPs": [{"ip": "10.1.90.166"}], "qosClass": "Burstable", "startTime": "2020-03-16T10:18:30Z"}}]}

TASK [tower : Set the tower pod name as a variable.] ***************************
task path: /opt/ansible/roles/tower/tasks/main.yml:26
ok: [localhost] => {"ansible_facts": {"tower_pod_name": "awx-tower-web-5b7bc9f756-8zlhh"}, "changed": false}

TASK [tower : Verify tower_pod_name is populated.] *****************************
task path: /opt/ansible/roles/tower/tasks/main.yml:30
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [tower : Check if database is populated (auth_user table exists).] ********
task path: /opt/ansible/roles/tower/tasks/main.yml:35
ok: [localhost] => {"changed": false, "cmd": "kubectl exec -n awx awx-tower-web-5b7bc9f756-8zlhh -- bash -c \"echo 'from django.db import connection; tbl = \\\"auth_user\\\" in connection.introspection.table_names(); exit(0 if tbl else 1)' | awx-manage shell\"", "delta": "0:00:04.092894", "end": "2020-03-16 10:22:49.651612", "rc": 0, "start": "2020-03-16 10:22:45.558718", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [tower : Migrate the database if the K8s resources were updated.] *********
task path: /opt/ansible/roles/tower/tasks/main.yml:47
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [tower : include_tasks] ***************************************************
task path: /opt/ansible/roles/tower/tasks/main.yml:54
[WARNING]: While constructing a mapping from
/opt/ansible/roles/tower/tasks/initialize.yml, line 2, column 3, found a
duplicate dict key (changed_when). Using last defined value only.
included: /opt/ansible/roles/tower/tasks/initialize.yml for localhost

TASK [tower : Check if there are any Tower super users defined.] ***************
task path: /opt/ansible/roles/tower/tasks/initialize.yml:2
ok: [localhost] => {"changed": false, "cmd": "kubectl exec -n awx awx-tower-web-5b7bc9f756-8zlhh -- bash -c \"echo 'from django.contrib.auth.models import User; nsu = User.objects.filter(is_superuser=True).count(); exit(0 if nsu > 0 else 1)' | awx-manage shell\"", "delta": "0:00:04.139437", "end": "2020-03-16 10:22:56.131640", "rc": 0, "start": "2020-03-16 10:22:51.992203", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [tower : Create Tower super user via Django if it doesn't exist.] *********
task path: /opt/ansible/roles/tower/tasks/initialize.yml:14
skipping: [localhost] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

TASK [tower : Create preload data if necessary.] *******************************
task path: /opt/ansible/roles/tower/tasks/initialize.yml:23
ok: [localhost] => {"changed": false, "cmd": "kubectl exec -n awx awx-tower-web-5b7bc9f756-8zlhh -- bash -c \"awx-manage create_preload_data\"", "delta": "0:00:05.259135", "end": "2020-03-16 10:23:03.636847", "rc": 0, "start": "2020-03-16 10:22:58.377712", "stderr": "", "stderr_lines": [], "stdout": "An organization is already in the system, exiting.\n(changed: False)", "stdout_lines": ["An organization is already in the system, exiting.", "(changed: False)"]}

TASK [tower : Scale the tower_task deployment to 0 replicas after migration.] ***
task path: /opt/ansible/roles/tower/tasks/main.yml:56
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [tower : Scale the tower_task deployment back to 1 replica after migration.] ***
task path: /opt/ansible/roles/tower/tasks/main.yml:63
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************
localhost                  : ok=8    changed=0    unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

[nicolaibaralmueller]

@nicolaibaralmueller Can you provide the logs from the ansible container of the operator (i.e. kubectl logs -c ansible)? I'm seeing something similar and I noticed this in my logs:

failed: [localhost] (item=tower_web.yaml.j2) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": "tower_web.yaml.j2", "msg": "Failed to create object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"ingresses.extensions \\\\\"ta-tower-ingress\\\\\" is invalid: spec.rules[0]: Forbidden: you do not have permission to set host fields in ingress rules\",\"reason\":\"Invalid\",\"details\":{\"name\":\"ta-tower-ingress\",\"group\":\"extensions\",\"kind\":\"ingresses\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: you do not have permission to set host fields in ingress rules\",\"field\":\"spec.rules[0]\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}

Because of this the play fails and doesn't get past processing the templates

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0 

And for that reason, I don't think the database setup is ever actually run (which is why it stays in updating/upgrading. Can't say that this will be your exact scenario, but something to look into.

cc/ @geerlingguy Interested on whether this is by design or if this is something that needs accounted for.

Hi @tylerauerbeck

Added --allow-privileged into /var/snap/microk8s/current/args/kube-apiserver and the error went away. I am not at the logon screen :)

[nicolaibaralmueller]

Delete namespace. Add --allow-privileged into /var/snap/microk8s/current/args/kube-apiserver followed by microk8s.stop and microk8s start. Create namespace and apply operator.