krisayala
commented
6 years ago I am seeing the same issues on Kuberentes version Major:"1", Minor:"10", GitVersion:"v1.10.4" and AWX 1.0.6 with the latest web and task images running on centOS 7.
However looking at my deployment file on Kubernetes I am seeing the following env variables under the awx-web container populated with the correct values.
kubectl edit deployment -n awx awx
- env:
- name: DATABASE_USER
value: <>
- name: DATABASE_NAME
value: <>
- name: DATABASE_HOST
value: <>
- name: DATABASE_PORT
value: "5432"
- name: DATABASE_PASSWORD
value: <>
- name: MEMCACHED_HOST
value: <>
- name: RABBITMQ_HOST
value: <>
- name: AWX_ADMIN_USER
value: <>
- name: AWX_ADMIN_PASSWORD
value: <>[VM ~]# kubectl logs -n awx awx-58bb76dc54-gttbm awx-web
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"changed": false,
"msg": "argument port is of type <type 'str'> and we were unable to convert to int: invalid literal for int() with base 10: ''"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"changed": false,
"elapsed": 300,
"msg": "Timeout when waiting for :11211"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"changed": false,
"elapsed": 300,
"msg": "Timeout when waiting for :5672"
}
lbrigmanArris
matburt
ISSUE TYPE
COMPONENT NAME
SUMMARY
During installation of AWX in OpenShift, we observe the following errors from the awx-web container:
================================ Because of these errors the availability of the API and web interfaces are delayed at least 10 minutes
ENVIRONMENT
Server https://gsm-t1.mdc.usaorbea.lab:8443 openshift v3.9.0+ba7faec-1 kubernetes v1.9.1+a0ce1bc657
STEPS TO REPRODUCE
Install AWX via ansible installer onto Openshift, Once the playbook is complete examine the logs of the awx-web container. oc logs -f awx- -c awx-web
EXPECTED RESULTS
No errors expected. Start up time not delayed by 10 minutes.
ACTUAL RESULTS
The errors reported above are observed and must wait at least 10 minutes to use the API or web interfaces.
ADDITIONAL INFORMATION
Analysis: The file that contains the commands that produce the error is: awx/installer/roles/image_build/files/launch_awx.sh: Lines 8-11. These require environment variables to allow proper execution. Specifically these variables: DATABASE_HOST DATABASE_PORT MEMCACHED_HOST RABBITMQ_HOST DATABASE_USER DATABASE_NAME DATABASE_PASSWORD Attached is a full dump of the env for a running awx-web container within AWX pod running on OpenShift:
oc rsh -c awx-web awx-6947865db5-xjvkj sh-4.2$ env KEYCLOAK_OPENSHIFT_SERVICE_PORT_8443_TCP=8443 LICENSE_SERVICE_PORT=8080 AWX_RMQ_MGMT_PORT_15672_TCP_ADDR=172.30.238.103 GIT_SERVICE_HOST=172.30.214.17 KEYCLOAK_OPENSHIFT_PORT_8443_TCP_PORT=8443 GRAFANA_PORT=tcp://172.30.173.161:3000 HOSTNAME=awx-6947865db5-xjvkj GRAFANA_PORT_3000_TCP_PROTO=tcp AWX_WEB_SVC_PORT_8052_TCP_PROTO=tcp ETCD_PORT_2380_TCP_PORT=2380 LICENSE_PORT=tcp://172.30.67.207:8080 AWX_ETCD_PORT_4001_TCP_ADDR=172.30.33.200 AWX_ETCD_PORT_4001_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_PORT=443 ETCD_SERVICE_PORT_CLIENT=2379 KUBERNETES_PORT=tcp://172.30.0.1:443 KEYCLOAK_OPENSHIFT_PORT_8443_TCP_ADDR=172.30.154.38 TERM=xterm POSTGRESQL_PORT_5432_TCP_ADDR=172.30.102.94 KEYCLOAK_OPENSHIFT_PORT_8443_TCP=tcp://172.30.154.38:8443 POSTGRESQL_PORT=tcp://172.30.102.94:5432 POSTGRESQL_SERVICE_PORT_POSTGRESQL=5432 AWX_RMQ_MGMT_PORT_15672_TCP_PORT=15672 AWX_RMQ_MGMT_SERVICE_PORT_RMQMGMT=15672 GIT_PORT_8080_TCP=tcp://172.30.214.17:8080 ETCD_PORT_2380_TCP=tcp://172.30.58.148:2380 AWX_ETCD_SERVICE_HOST=172.30.33.200 KUBERNETES_SERVICE_PORT=443 KEYCLOAK_OPENSHIFT_PORT_8443_TCP_PROTO=tcp KEYCLOAK_OPENSHIFT_SERVICE_PORT=8080 GIT_PORT_8080_TCP_PORT=8080 KUBERNETES_SERVICE_HOST=172.30.0.1 KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53 GRAFANA_PORT_3000_TCP_ADDR=172.30.173.161 GIT_SERVICE_PORT=8080 AWX_WEB_SVC_PORT_8052_TCP_ADDR=172.30.141.236 POSTGRESQL_PORT_5432_TCP=tcp://172.30.102.94:5432 KUBERNETES_PORT_53_TCP_PORT=53 INFLUXDB_PORT_8086_TCP=tcp://172.30.55.129:8086 KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53 AWX_RMQ_MGMT_PORT_15672_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_DNS=53 LICENSE_PORT_8080_TCP_PORT=8080 INFLUXDB_SERVICE_HOST=172.30.55.129 KEYCLOAK_OPENSHIFT_PORT_8080_TCP=tcp://172.30.154.38:8080 ETCD_PORT_2379_TCP_PORT=2379 YUM_SERVER_PORT_8080_TCP_PORT=8080 KEYCLOAK_OPENSHIFT_PORT=tcp://172.30.154.38:8080 ETCD_PORT_2379_TCP_PROTO=tcp AWX_ETCD_SERVICE_PORT_AWX_ETCD=4001 YUM_SERVER_PORT_8080_TCP_PROTO=tcp GRAFANA_SERVICE_PORT=3000 LICENSE_PORT_8080_TCP_ADDR=172.30.67.207 GIT_PORT_8080_TCP_PROTO=tcp KEYCLOAK_OPENSHIFT_PORT_8080_TCP_PROTO=tcp KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1 AWX_WEB_SVC_SERVICE_HOST=172.30.141.236 GRAFANA_PORT_3000_TCP_PORT=3000 GIT_PORT=tcp://172.30.214.17:8080 AWX_WEB_SVC_PORT_8052_TCP_PORT=8052 POSTGRESQL_SERVICE_HOST=172.30.102.94 KEYCLOAK_OPENSHIFT_SERVICE_PORT_8080_TCP=8080 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1 INFLUXDB_PORT_8086_TCP_ADDR=172.30.55.129 AWX_WEB_SVC_SERVICE_PORT_HTTP=8052 AWX_ETCD_PORT_4001_TCPPORT=4001 =/usr/bin/env AWX_RMQ_MGMT_SERVICE_PORT=15672 AWX_RMQ_MGMT_PORT=tcp://172.30.238.103:15672 YUM_SERVER_SERVICE_PORT=8080 YUM_SERVER_PORT=tcp://172.30.200.40:8080 AWX_ETCD_PORT=tcp://172.30.33.200:4001 PWD=/var/lib/awx AWX_WEB_SVC_PORT=tcp://172.30.141.236:8052 AWX_RMQ_MGMT_SERVICE_HOST=172.30.238.103 YUM_SERVER_SERVICE_HOST=172.30.200.40 ETCD_PORT_2380_TCP_ADDR=172.30.58.148 INFLUXDB_PORT_8086_TCP_PORT=8086 ETCD_PORT_2380_TCP_PROTO=tcp KUBERNETES_PORT_53_UDP_PORT=53 KEYCLOAK_OPENSHIFT_SERVICE_HOST=172.30.154.38 INFLUXDB_SERVICE_PORT_8086_TCP=8086 ETCD_SERVICE_PORT_SERVER=2380 POSTGRESQL_PORT_5432_TCP_PORT=5432 KEYCLOAK_OPENSHIFT_PORT_8080_TCP_ADDR=172.30.154.38 HOME=/var/lib/awx SHLVL=2 INFLUXDB_PORT_8086_TCP_PROTO=tcp KUBERNETES_PORT_53_UDP_PROTO=udp KUBERNETES_PORT_443_TCP_PROTO=tcp POSTGRESQL_SERVICE_PORT=5432 GRAFANA_PORT_3000_TCP=tcp://172.30.173.161:3000 INFLUXDB_PORT=tcp://172.30.55.129:8086 KUBERNETES_SERVICE_PORT_HTTPS=443 AWX_WEB_SVC_PORT_8052_TCP=tcp://172.30.141.236:8052 GRAFANA_SERVICE_PORT_3000_TCP=3000 ETCD_SERVICE_PORT=2379 ETCD_PORT_2379_TCP_ADDR=172.30.58.148 YUM_SERVER_PORT_8080_TCP_ADDR=172.30.200.40 POSTGRESQL_PORT_5432_TCP_PROTO=tcp AWX_RMQ_MGMT_PORT_15672_TCP=tcp://172.30.238.103:15672 GIT_PORT_8080_TCP_ADDR=172.30.214.17 AWX_WEB_SVC_SERVICE_PORT=8052 LICENSE_PORT_8080_TCP=tcp://172.30.67.207:8080 INFLUXDB_SERVICE_PORT=8086 GRAFANA_SERVICE_HOST=172.30.173.161 AWX_ETCD_PORT_4001_TCP=tcp://172.30.33.200:4001 ETCD_PORT_2379_TCP=tcp://172.30.58.148:2379 KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1 KUBERNETES_PORT_53_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_DNS_TCP=53 YUM_SERVER_SERVICE_PORT_8080_TCP=8080 YUM_SERVER_PORT_8080_TCP=tcp://172.30.200.40:8080 LICENSE_PORT_8080_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443 ETCD_PORT=tcp://172.30.58.148:2379 AWX_ETCD_SERVICE_PORT=4001 LICENSE_SERVICE_HOST=172.30.67.207 ETCD_SERVICE_HOST=172.30.58.148 KEYCLOAK_OPENSHIFT_PORT_8080_TCP_PORT=8080
======================================== The required variables are not in the environment. Similar variables are available but are not being used in the script.
In the file: awx/installer/roles/kubernetes/templates/deployment.yml.j2 The variables that are needed are provided to the awx-celery (lines 151-165). There are no environment variables provided to awx-web container.
The solution: Duplicate the environment variables to the awx-web container or create a new config map for both containers to have these environment variables.