search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.94k stars 3.41k forks source link

Error starting project unknown cause #3956

Open ssaammoo opened 5 years ago

ssaammoo commented 5 years ago
ISSUE TYPE
SUMMARY

AWX Installation failed with unknow error

ENVIRONMENT
STEPS TO REPRODUCE

I had AWX 3.0.1 running and then I tried to install AWX 4.0.0 on top, the same procedure worked fine to upgrade AWX 3.0.0. to AWX 3.0.1

`# ansible-playbook -i inventory install.yml -vvvv

.
.
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx-4/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274 `" && echo ansible-tmp-1559049469.25-220943567830274="` echo /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-60293Nj0BSi/tmpiVJnh7 TO /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/ /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_docker_service_payload_Dj9vLs/__main__.py", line 764, in cmd_up
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
    'Encountered errors while bringing up the project.'

[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
    "changed": false,
    "errors": [],
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "project_name": null,
            "project_src": "/var/lib/awx",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": 10,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
    "module_stdout": "",
    "msg": "Error starting project unknown cause"
}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=8    changed=0    unreachable=0    failed=1    skipped=77   rescued=0    ignored=0

`

more dockers are created:

`# docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
6b20e23a0be7        ansible/awx_rabbitmq:3.7.4   "docker-entrypoint.s…"   7 minutes ago       Up 6 seconds        4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   awx_rabbitmq
5d25e036cd96        memcached:alpine             "docker-entrypoint.s…"   7 minutes ago       Up 6 seconds        11211/tcp                                             awx_memcached
695c8aa6d2b7        postgres:9.6                 "docker-entrypoint.s…"   7 minutes ago       Up 5 seconds        5432/tcp                                              awx_postgres
8246270599a5        ansible/awx_task:3.0.1       "/tini -- /bin/sh -c…"   About an hour ago   Up 4 seconds        8052/tcp                                              awx_task
cb18997147ae        ansible/awx_web:3.0.1        "/tini -- /bin/sh -c…"   About an hour ago   Up 4 seconds        0.0.0.0:8052->8052/tcp                                awx_web
1580ef93f47d        memcached:alpine             "docker-entrypoint.s…"   21 hours ago        Up 6 seconds        11211/tcp                                             memcached
97c10bb0b7ac        ansible/awx_rabbitmq:3.7.4   "docker-entrypoint.s…"   21 hours ago        Up 6 seconds        4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   rabbitmq
4c527a00794b        postgres:9.6                 "docker-entrypoint.s…"   21 hours ago        Up 6 seconds        5432/tcp                                              postgres`
EXPECTED RESULTS

AWX Installed successfully

ACTUAL RESULTS

    "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
    "module_stdout": "",
    "msg": "Error starting project unknown cause"
ADDITIONAL INFORMATION
`# cat inventory
localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python"

[all:vars]

# Remove these lines if you want to run a local image build
# Otherwise the setup playbook will install the official Ansible images. Versions may
# be selected based on: latest, 1, 1.0, 1.0.0, 1.0.0.123
# by default the base will be used to search for ansible/awx_web and ansible/awx_task
dockerhub_base=ansible

# Openshift Install
# Will need to set -e openshift_password=developer -e docker_registry_password=$(oc whoami -t)
#           or set -e openshift_token=TOKEN
# openshift_host=127.0.0.1:8443
# openshift_project=awx
# openshift_user=developer
# openshift_skip_tls_verify=False
# openshift_pg_emptydir=True

# Kubernetes Install
# kubernetes_context=test-cluster
# kubernetes_namespace=awx
# tiller_namespace=kube-system
# Optional Kubernetes Variables
# pg_persistence_storageClass=StorageClassName
# pg_cpu_limit=1000
# pg_mem_limit=2

# Kubernetes and Openshift Install Resource Requests
# This are the request and limit values for a pod's container for task/web/rabbitmq/memcached/management.
# The total amount of requested resources for a pod is the sum of all
# resources requested by all containers in the pod
# A cpu_request of 1500 is 1.5 cores for the container to start out with.
# A cpu_limit defines the maximum cores that that container can reserve.
# A mem_request of 2 is for 2 gigabytes of memory for the container
# A mem_limit defines the maximum memory that that container can reserve.
# Default values for these entries can be found in ./roles/kubernetes/defaults/main.yml
# task_cpu_request=1500
# task_mem_request=2
# task_cpu_limit=2000
# task_mem_limit=4
# web_cpu_limit=1000
# web_mem_limit=2
# rabbitmq_cpu_limit=1000
# rabbitmq_mem_limit=3
# memcached_cpu_limit=1000
# memcached_mem_limit=2
# management_cpu_limit=2000
# management_mem_limit=2

# Common Docker parameters
awx_task_hostname=awx
awx_web_hostname=awxweb
postgres_data_dir=/ansible/pgdocker
host_port=8052

# Docker Compose Install
# use_docker_compose=false
# The docker_compose.yml file will be created in this directory
# The name of the directory (here "awx") will be the prefix of the docker containers
docker_compose_dir=/var/lib/awx

# Required for Openshift when building the image on your own
# Optional for Openshift if using Dockerhub or another prebuilt registry
# Required for Standalone Docker Install if building the image on your own
# Optional for Standalone Docker Install if using Dockerhub or another prebuilt registry
# Required for Docker Compose Install if building the image on your own
# Optional for Docker Compose Install if using Dockerhub or another prebuilt registry
# Define if you want the image pushed to a registry. The container definition will also use these images
# docker_registry=172.30.1.1:5000
# docker_registry_repository=awx
# docker_registry_username=developer

# Docker_image will not attempt to push to remote if the image already exists locally
# Set this to true to delete images from docker on the build host so that they are pushed to the remote repository
# docker_remove_local_images=False

# Set pg_hostname if you have an external postgres server, otherwise
# a new postgres service will be created
# pg_hostname=postgresql
pg_username=xxxx
# pg_password should be random 10 character alphanumeric string, when postgresql is running on kubernetes
# NB: it's a limitation of the "official" postgres helm chart
pg_password=xxxx$$
pg_database=xxxx_db
pg_port="5432"

# RabbitMQ Configuration
rabbitmq_password=XXXX
rabbitmq_erlang_cookie=XXXX

# Use a local distribution build container image for building the AWX package
# This is helpful if you don't want to bother installing the build-time dependencies as
# it is taken care of already.
# NOTE: IMPORTANT: If you are running a mininshift install, using this container might not work
#                  if you are using certain drivers like KVM where the source tree can't be mapped
#                  into the build container.
#                  Thus this setting must be set to False which will trigger a local build. To view the
#                  typical dependencies that you might need to install see:
#                  installer/image_build/files/Dockerfile.sdist
# use_container_for_build=true

# This will create or update a default admin (superuser) account in AWX, if not provided
# then these default values are used
admin_user=admin
admin_password=XXXXX

# Whether or not to create preload data for demonstration purposes
create_preload_data=True

# AWX Secret key
# It's *very* important that this stay the same between upgrades or you will lose the ability to decrypt
# your credentials
secret_key=XXXXX

# Build AWX with official logos
# Requires cloning awx-logos repo into the project root.
# Review the trademark guidelines at https://github.com/ansible/awx-logos/blob/master/TRADEMARKS.md
# awx_official=false

# Proxy
#http_proxy=http://proxy:3128
#https_proxy=http://proxy:3128
#no_proxy=mycorp.org

# Container networking configuration
# Set the awx_task and awx_web containers' search domain(s)
#awx_container_search_domains=example.com,ansible.com
# Alternate DNS servers
#awx_alternate_dns_servers="10.1.2.3,10.2.3.4"

# AWX project data folder. If you need access to the location where AWX stores the projects
# it manages from the docker host, you can set this to turn it into a volume for the container.
#project_data_dir=/var/lib/awx/projects
project_data_dir=/ansible/projects

# CA Trust directory. If you need to provide custom CA certificates, supplying
# this variable causes this directory on the host to be bind mounted over
# /etc/pki/ca-trust in the awx_task and awx_web containers.
# NOTE: only obeyed in local_docker install
#ca_trust_dir=/etc/pki/ca-trust/source/anchors

# Include /etc/nginx/awx_extra.conf
# Note the use of glob pattern for nginx
# which makes include "optional" - i.e. not fail
# if file is absent
#extra_nginx_include="/etc/nginx/awx_extra[.]conf"`
ryanpetrello commented 5 years ago

@ssaammoo I've seen errors like this with older docker-compose versions. What version of docker-compose are you using? Try updating to the latest?

ssaammoo commented 5 years ago

@ryanpetrello here we go:

# docker-compose --version
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)
docker-compose version 1.24.0, build 0aa5906
ryanpetrello commented 5 years ago

@ssaammoo this version is working for me, maybe give it a shot?

docker-compose version 1.23.2, build 1110ad0
ssaammoo commented 5 years ago

Unfortunately didn't work either :-(

TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx-4/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879 `" && echo ansible-tmp-1559123810.83-41538014110879="` echo /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-72810XBXq4/tmpTRtFv8 TO /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/ /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_docker_service_payload_JsXP6k/__main__.py", line 764, in cmd_up
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/compose/project.py", line 507, in up
    'Encountered errors while bringing up the project.'

[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
    "changed": false,
    "errors": [],
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "project_name": null,
            "project_src": "/var/lib/awx",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": 10,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
    "module_stdout": "",
    "msg": "Error starting project unknown cause"
}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=8    changed=0    unreachable=0    failed=1    skipped=77   rescued=0    ignored=0

and got multiple dockers

[root@alva-atmans01 installer]# docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
8acb85229d6c        memcached:alpine             "docker-entrypoint.s…"   24 seconds ago      Up 23 seconds       11211/tcp                                             awx_memcached
1465ade16aba        ansible/awx_rabbitmq:3.7.4   "docker-entrypoint.s…"   24 seconds ago      Up 23 seconds       4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   awx_rabbitmq
adef4b3ff855        postgres:9.6                 "docker-entrypoint.s…"   24 seconds ago      Up 23 seconds       5432/tcp                                              awx_postgres
8246270599a5        ansible/awx_task:3.0.1       "/tini -- /bin/sh -c…"   22 hours ago        Up 18 hours         8052/tcp                                              awx_task
cb18997147ae        ansible/awx_web:3.0.1        "/tini -- /bin/sh -c…"   22 hours ago        Up 18 hours         0.0.0.0:8052->8052/tcp                                awx_web
1580ef93f47d        memcached:alpine             "docker-entrypoint.s…"   42 hours ago        Up 18 hours         11211/tcp                                             memcached
97c10bb0b7ac        ansible/awx_rabbitmq:3.7.4   "docker-entrypoint.s…"   42 hours ago        Up 18 hours         4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   rabbitmq
4c527a00794b        postgres:9.6                 "docker-entrypoint.s…"   42 hours ago        Up 18 hours         5432/tcp                                              postgres
[root@alva-atmans01 installer]#
picapdjoe commented 5 years ago

Regretfully I'm running into the same thing. If you attempt to start the containers using docker-compose or docker directly, you'll get the following:

Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/awxcompose/SECRET_KEY\\\" to rootfs \\\"/var/lib/docker/overlay2/dcb48bab9cd37651132f72a37c17d1450940272cff5cec20c85544cb04d33c48/merged\\\" at \\\"/var/lib/docker/overlay2/dcb48bab9cd37651132f72a37c17d1450940272cff5cec20c85544cb04d33c48/merged/etc/tower/SECRET_KEY\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

If you attempt the following, you'll get the error specified by the OP

$ sudo ansible-playbook -i inventory install.yml

Looking at where the installer is pulling the files, SECRET_KEY exists and is a directory:

$ ls -lah /tmp/awxcompose/
total 28K
drwxr-xr-x  5 root root 4.0K May 29 12:00 .
drwxrwxrwt 10 root root 4.0K May 29 12:21 ..
drwxr-xr-x  2 root root 4.0K May 29 08:56 credentials.py
-rw-------  1 root root 1.8K May 29 08:56 docker-compose.yml
drwxr-xr-x  2 root root 4.0K May 29 08:56 environment.sh
drwxr-xr-x  2 root root 4.0K May 29 08:50 SECRET_KEY

Ubuntu 16.04 docker-compose 1.24.0 docker 18.09

kernelzilla commented 5 years ago

FWIW I encountered the same error when starting awx-web which wants to listen on port 80, turned out a separate install of nginx was using it (confirm via netstat -anpo | grep 80). After stopping nginx, the install playbook completed successfully.

kernelzilla commented 5 years ago

I've opened a pull request (#4002) to check and give the user a clearer message regarding the port conflict instead of having to check awx_web log status for the error.

ryanpetrello commented 5 years ago

@ssaammoo @picapdjoe any chance either of you are encountering something similar?

jomach commented 5 years ago

I had the same issue and it is port conflict.

ssaammoo commented 5 years ago

@ryanpetrello Hi Ryan, I am redoing the test now with a new version of AWX (5.0), I stopped nginx that was running as reverse proxy so I won't get a conflict, however I am still getting the same issue:

TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945 `" && echo ansible-tmp-1562668950.98-226850187534945="` echo /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-48857My9OcO/tmpMYu8nE TO /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/ /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_docker_service_payload_XzUYXM/__main__.py", line 764, in cmd_up
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
    'Encountered errors while bringing up the project.'

[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
    "changed": false,
    "errors": [],
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "project_name": null,
            "project_src": "/tmp/awxcompose",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": 10,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
    "module_stdout": "",
    "msg": "Error starting project unknown cause"
}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=8    changed=0    unreachable=0    failed=1    skipped=77   rescued=0    ignored=0
ssaammoo commented 5 years ago

I have just tried with AWX 6.0 and got the same result :-(

TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105 `" && echo ansible-tmp-1562670134.55-11038997918105="` echo /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-20508258sWX/tmpTqiJI7 TO /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/ /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_docker_service_payload_xgDsOe/__main__.py", line 764, in cmd_up
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
    'Encountered errors while bringing up the project.'

[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
    "changed": false,
    "errors": [],
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "project_name": null,
            "project_src": "/tmp/awxcompose",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": 10,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
    "module_stdout": "",
    "msg": "Error starting project unknown cause"
}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=8    changed=0    unreachable=0    failed=1    skipped=77   rescued=0    ignored=0
ssaammoo commented 5 years ago

I think my problem is not related to the port (at least now), I have add check port to my playbook: https://github.com/kernelzilla/awx/commit/949e37e809513adf66ba868d9c3bf5088696ec2e And I am getting still generic error.

TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {"changed": false, "errors": [], "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B", "module_stdout": "", "msg": "Error starting project unknown cause"}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=10   changed=1    unreachable=0    failed=1    skipped=79   rescued=0    ignored=0

If I start the nginx I get error about the port, so my problem now is something else (I am trying with AWX 6.0)

# systemctl start nginx
# ansible-playbook -i inventory install.yml | tail -25

TASK [check_vars : include_tasks] ***********************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [check_vars : include_tasks] ***********************************************************************************************************************************************************************************************************
included: /root/awx/installer/roles/check_vars/tasks/check_docker.yml for localhost

TASK [check_vars : postgres_data_dir should be defined] *************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [check_vars : host_port should be defined] *********************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [check_vars : host_port should be available] *******************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "elapsed": 2, "msg": "host_port is currently in use"}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
ssaammoo commented 5 years ago

Can someone please tell me how can I check the logs of the installation

ssaammoo commented 5 years ago

anyone can help me please :-)

ssaammoo commented 5 years ago

So I uninstall NGINX from the machine, rebooted it and installed available CentOS updates, I got the AWX ver 6 again and I tried to update again, I got same error:

TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101 `" && echo ansible-tmp-1563197952.57-243929466938101="` echo /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-33608Frek_n/tmpLy2Mno TO /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/ /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_docker_service_payload_rbsc6r/__main__.py", line 764, in cmd_up
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
    'Encountered errors while bringing up the project.'

[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
    "changed": false,
    "errors": [],
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "project_name": null,
            "project_src": "/tmp/awxcompose",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": 10,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
    "module_stdout": "",
    "msg": "Error starting project unknown cause"
}
ssaammoo commented 5 years ago

I disable nginx and I tried to upgrade from 3.1 to 6.1 and still getting the same error ...the logs are not guiding to certain error and doesn't seem to be port thing:

TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {"changed": false, "errors": [], "module_stderr": "Creating awx_postgres ... \r\nCreating awx_rabbitmq ... \r\nCreating awx_memcached ... \r\n\r\u001b[2BCreating awx_web       ... \r\n\r\u001b[1B", "module_stdout": "6.1.0: Pulling from ansible/awx_web\nDigest: sha256:51a4f11acc4b1f47a1ed233f866035246925781b997937709f64c69863f993f4\nStatus: Downloaded newer image for ansible/awx_web:6.1.0\n6.1.0: Pulling from ansible/awx_task\nDigest: sha256:3044edfcaf9924b488632750c65d41b8fb54058aee0018291b9aac0fdb7fcc67\nStatus: Downloaded newer image for ansible/awx_task:6.1.0\n", "msg": "Error starting project unknown cause"}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=8    changed=1    unreachable=0    failed=1    skipped=77   rescued=0    ignored=0

# netstat -an | grep "80\|443"
tcp        0      0 10.14.15.60:51836       10.14.11.2:8080         TIME_WAIT
tcp        0      0 10.14.15.60:51842       10.14.11.2:8080         TIME_WAIT
tcp        0      0 10.14.15.60:51844       10.14.11.2:8080         TIME_WAIT
tcp        0      0 10.14.15.60:51838       10.14.11.2:8080         TIME_WAIT
tcp        0      0 10.14.15.60:51840       10.14.11.2:8080         TIME_WAIT
tcp        0      0 10.14.15.60:48098       10.14.15.194:58547      ESTABLISHED
tcp        0      0 10.14.15.60:51846       10.14.11.2:8080         TIME_WAIT
tcp        0      0 10.14.15.60:51848       10.14.11.2:8080         TIME_WAIT
tcp6       0      0 :::8052                 :::*                    LISTEN
udp6       0      0 fe80::c485:21ff:fec:123 :::*
udp6       0      0 fe80::42:e1ff:feb2::123 :::*
udp6       0      0 fe80::706a:29ff:fe3:123 :::*
udp6       0      0 fe80::c0ae:63ff:fef:123 :::*
udp6       0      0 fe80::6c2a:80ff:feb:123 :::*
udp6       0      0 fe80::48d2:69ff:fe5:123 :::*
udp6       0      0 fe80::783b:7bff:fe7:123 :::*
udp6       0      0 fe80::60de:43ff:fe2:123 :::*
udp6       0      0 fe80::f402:40ff:fe8:123 :::*
udp6       0      0 fe80::42:55ff:fe71::123 :::*
udp6       0      0 fe80::250:56ff:fe92:123 :::*
unix  2      [ ACC ]     STREAM     LISTENING     34080    private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     13307    @/containerd-shim/moby/4c527a00794bd9fc0ad498a50d458c81828082714ee4f4329e9a635caa511b99/shim.sock
unix  2      [ ACC ]     STREAM     LISTENING     91455    @/containerd-shim/moby/7007a14577d1ac0d20053787c08237f6d3c1dbf223f8f80c69ec53c693a7fc22/shim.sock
unix  2      [ ACC ]     STREAM     LISTENING     38950    @/containerd-shim/moby/1580ef93f47d99a68cc7d2abd13f302db9360a7d80b7f1436578065795d7ba30/shim.sock
unix  3      [ ]         STREAM     CONNECTED     13180
unix  3      [ ]         STREAM     CONNECTED     980      /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     36010    @/containerd-shim/moby/1580ef93f47d99a68cc7d2abd13f302db9360a7d80b7f1436578065795d7ba30/shim.sock
unix  3      [ ]         STREAM     CONNECTED     36027    @/containerd-shim/moby/4c527a00794bd9fc0ad498a50d458c81828082714ee4f4329e9a635caa511b99/shim.sock
unix  3      [ ]         STREAM     CONNECTED     92938    @/containerd-shim/moby/7007a14577d1ac0d20053787c08237f6d3c1dbf223f8f80c69ec53c693a7fc22/shim.sock
unix  2      [ ]         DGRAM                    80797
konkolorado commented 5 years ago

Are there any stopped containers after the task fails whose logs you can see? If there are, doing a docker logs awx_web might have more information on what's going wrong.

File "/tmp/ansible_docker_service_payload_rbsc6r/main.py", line 764, in cmd_up timeout=self.timeout) File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up 'Encountered errors while bringing up the project.'

This makes me think something's up with your Docker daemon. Does AWX start if you do a docker-compose up -d in the /tmp/awxcompose/ directory?

zbenta commented 5 years ago

I'm having the same issue. When I do the docker-compose up -d I get the following message:

awx_postgres is up-to-date awx_rabbitmq is up-to-date awx_memcached is up-to-date

Starting 02c59e00ac79_awx_web ... error

ERROR: for 02c59e00ac79_awx_web Cannot start service web: driver failed programming >external connectivity on endpoint 02c59e00ac79_awx_web >(0944c6ee2f7faecd88f5c38804b022087830a28c6daa846ab11d0fcf502a30ae): (iptables failed: i> iptables --wait -t filter -A DOCKER ! -i br-XX -o br-XX -p tcp -d 172.18.0.5 >--dport 8052 -j ACCEPT: iptables: No chain/target/match by that name. (exit status 1))

ERROR: for web Cannot start service web: driver failed programming external connectivity on >endpoint 02c59e00ac79_awx_web >(0944c6ee2f7faecd88f5c38804b022087830a28c6daa846ab11d0fcf502a30ae): (iptables failed: >iptables --wait -t filter -A DOCKER ! -i br-XX -o br-XX -p tcp -d 172.18.0.5 >--dport 8052 -j ACCEPT: iptables: No chain/target/match by that name. (exit status 1)) ERROR: Encountered errors while bringing up the project.

I've added the following rule to my iptables script: -A INPUT -s x.x.x.x/xx -p tcp --dport 8052 -j ACCEPT But still I can't start the project.

zbenta commented 5 years ago

The Strangest thing happened, I updated my OS, rebooted the machine and it started correctly. Go figure...

doves99 commented 5 years ago

Have you tried, removing the docker images and re-running the install command?. Hope you backed up the postgres/playbooks.

tchiapuziowong commented 5 years ago

When I try to create the docker images manually I get the following error, I've already killed any existing containers before running:

administrator@admin-awx-ubuntu:/tmp/awxcompose$ docker-compose up -d
Starting awx_memcached ... done
Starting awx_postgres  ... done
Starting awx_rabbitmq  ... done
Creating awx_web       ... error

ERROR: for awx_web  Cannot create container for service web: Conflict. The container name "/awx_web" is already in use by container "187153c1ad90f43db84544cf343c87148b94557d238637529deac95136ec9e43". You have to remove (or rename) that container to be able to reuse that name.

ERROR: for web  Cannot create container for service web: Conflict. The container name "/awx_web" is already in use by container "187153c1ad90f43db84544cf343c87148b94557d238637529deac95136ec9e43". You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.
administrator@admin-awx-ubuntu:/tmp/awxcompose$
ssbarnea commented 5 years ago

I am facing the same confusing error in a totally unrelated place. This bug is caused by https://github.com/docker/compose/issues/6899 which prevents API users (like ansible module) from getting the real error that caused the failure.

Based on previous experience it will take a lot ok poking to make Docker fix this bug, but the sooner we start the faster we are likely to address it.

konkolorado commented 5 years ago

Just ran into this. I had to remove and deploy from scratch. Why this happens originally, I don't know.

memelet commented 5 years ago

I see this error if any containers of the same still exists (ie, docker ps -a). Manually removing the containers gets around the "error".

ryanpetrello commented 5 years ago

What most of you are encountering here does look to me like a symptom of a bug in docker-compose itself: https://github.com/docker/compose/issues/6899

ssbarnea commented 5 years ago

Even after following steps mentioned by @konkolorado I am facing the same issue.

gcgoncalves commented 4 years ago

I'm getting this error as well. A temporary workaround that has been working for me is to checkout commit 44907b33dc17b528dfd1d6dfa2c990c2d31d307a before installing. any commit more recent than this one will fail with unknown error on my installation attempts.

brbrow commented 4 years ago

I was facing the same problem. Started from scrath and isntalled ansible using PIP to force it to use python3 I created this tutorial.

vincentkersten commented 4 years ago

@zbenta thanks: When I do the docker-compose up -d helped me to determine the issue:

ERROR: for awx_web  Cannot create container for service web: b'invalid volume specification:
 \'/Users/vincent/Desktop/AWX/opt/my-envs:~/Desktop/AWX/opt/my-envs:rw\': invalid mount 
config for type "bind": invalid mount path: \'~/Desktop/AWX/opt/my-envs\' mount path must be 
absolute'

setting the custom_venv_dir="/Users/vincent/Desktop/AWX/opt/my-envs/" as an absolute path in the file install/inventory and not "'~/Desktop/AWX/opt/my-envs" made all the issues go away :)

sivabgl commented 4 years ago

change it in the inventory file

ansible/awx_tasksud

dockerhub_base=ansible

will work

isweluiz commented 2 years ago

SOLVED BY MY SIDE. Hey guys, let me share something with you. some important points to investigate on your side:

If you have some avahi process running from tower(before installation): 

ps aux | grep avahi | awk  '{print $2}'| xargs kill -9

To kill all containers and then delete all images: 

docker stop $(docker ps -q)  ; docker rm $(docker ps -a)  
docker rmi $(docker images)

When you will run the playbook, use the verbose mode to have more details about the error -vvv

First error to me:

The full traceback is:
  File "/tmp/ansible_docker_compose_payload_owsfbjvw/ansible_docker_compose_payload.zip/ansible/modules/cloud/docker/docker_compose.py", line 791, in cmd_up
  File "/usr/local/lib/python3.6/site-packages/compose/project.py", line 507, in up
    'Encountered errors while bringing up the project.'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "errors": [],
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "project_name": null,
            "project_src": "/root/.awx/awxcompose",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": true,
            "timeout": 10,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "module_stderr": "Starting awx_web ... \n\u001b[1A\u001b[2K\nStarting awx_web ... \n\u001b[1B",
    "module_stdout": "",
    "msg": "Error starting project Encountered errors while bringing up the project."
}

Second Error

fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "docker-compose run --rm --service-ports task awx-manage migrate --no-input",
    "delta": "0:00:53.342387",
    "end": "2021-12-11 19:02:31.300163",
    "invocation": {
        "module_args": {
            "_raw_params": "docker-compose run --rm --service-ports task awx-manage migrate --no-input",
            "_uses_shell": true,
            "argv": null,
            "chdir": "/root/.awx/awxcompose",
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2021-12-11 19:01:37.957776",
    "stderr": "Creating network \"awxcompose_default\" with the default driver\nPulling postgres (postgres:12)...\nPulling web (ansible/awx:17.1.0)...\nfailed to register layer: Error processing tar file(exit status 1): open /var/lib/awx/venv/awx/lib/python3.6/site-packages/aiohttp/__pycache__/web_response.cpython-36.pyc: no space left on device",
    "stderr_lines": [
        "Creating network \"awxcompose_default\" with the default driver",
        "Pulling postgres (postgres:12)...",
        "Pulling web (ansible/awx:17.1.0)...",
        "failed to register layer: Error processing tar file(exit status 1): open /var/lib/awx/venv/awx/lib/python3.6/site-packages/aiohttp/__pycache__/web_response.cpython-36.pyc: no space left on device"
    ],
    "stdout": "12: Pulling from library/postgres\n17.1.0: Pulling from ansible/awx",
    "stdout_lines": [
        "12: Pulling from library/postgres",
        "17.1.0: Pulling from ansible/awx"
    ]
}

Success

After cleaning all and allowing more resources on my lab machine disk.

$ ansible-playbook -i installer/inventory installer/install.yml -vvvv

ok: [localhost] => {
    "changed": false,
    "cmd": [
        "docker",
        "exec",
        "awx_task",
        "bash",
        "-c",
        "/usr/bin/awx-manage create_preload_data"
    ],
    "delta": "0:00:12.361345",
    "end": "2021-12-11 19:09:19.735999",
    "invocation": {
        "module_args": {
            "_raw_params": "docker exec awx_task bash -c \"/usr/bin/awx-manage create_preload_data\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2021-12-11 19:09:07.374654",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "An organization is already in the system, exiting.\n(changed: False)",
    "stdout_lines": [
        "An organization is already in the system, exiting.",
        "(changed: False)"
    ]
}
META: ran handlers
META: ran handlers

PLAY RECAP *****************************************************************************************************
localhost                  : ok=21   changed=7    unreachable=0    failed=0    skipped=73   rescued=0    ignored=1   
docker ps
CONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS          PORTS                                   NAMES
e09076cc2e5e   ansible/awx:17.1.0   "/usr/bin/tini -- /u…"   14 minutes ago   Up 13 minutes   8052/tcp                                awx_task
c0534310a775   ansible/awx:17.1.0   "/usr/bin/tini -- /b…"   14 minutes ago   Up 13 minutes   0.0.0.0:80->8052/tcp, :::80->8052/tcp   awx_web
e2ecd09dbcc1   redis                "docker-entrypoint.s…"   14 minutes ago   Up 13 minutes   6379/tcp                                awx_redis
027c7bf28756   postgres:12          "docker-entrypoint.s…"   14 minutes ago   Up 13 minutes   5432/tcp                                awx_postgres

curl 192.168.99.110 -LI
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 11 Dec 2021 22:26:09 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1082
Connection: keep-alive
Vary: Accept-Language, Origin, Cookie
Content-Language: en
X-API-Total-Time: 0.019s
Strict-Transport-Security: max-age=15768000
X-Frame-Options: DENY