[Docker] make docker-compose-build fails with: EOF

[Commifreak]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

During make docker-compose-build I get: ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

I work behind proxy, but env variables are set (system, docker daemon and docker build config). What else can I try?

AWX version

21.11.0

Select the relevant components

• [ ] UI
• [ ] API
• [ ] Docs
• [ ] Collection
• [ ] CLI
• [X] Other

Installation method

docker development environment

Modifications

no

Ansible version

2.10.8

Operating system

Ubuntu 22.04

Web browser

No response

Steps to reproduce

build docker image

Expected results

no error

Actual results

EOF error

Additional information

~/awx$ make docker-compose-build
ansible-playbook tools/ansible/dockerfile.yml -e build_dev=True -e receptor_image=quay.io/ansible/receptor:latest
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Render AWX Dockerfile and sources] **********************************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [dockerfile : Create _build directory] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [dockerfile : Render supervisor configs] *****************************************************************************************************************************************************************
ok: [localhost] => (item=supervisor.conf)
ok: [localhost] => (item=supervisor_task.conf)

TASK [dockerfile : Render Dockerfile] *************************************************************************************************************************************************************************
ok: [localhost]

PLAY RECAP ****************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

DOCKER_BUILDKIT=1 docker build -t ghcr.io/ansible/awx_devel:devel \
    --build-arg BUILDKIT_INLINE_CACHE=1 \
    --cache-from=ghcr.io/ansible/awx_devel:devel .
[+] Building 324.5s (49/50)
 => [internal] load build definition from Dockerfile                                                                                                                                                      0.0s
 => => transferring dockerfile: 7.65kB                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                         0.0s
 => => transferring context: 56B                                                                                                                                                                          0.0s
 => [internal] load metadata for quay.io/ansible/receptor:latest                                                                                                                                          1.0s
 => [internal] load metadata for quay.io/centos/centos:stream9                                                                                                                                            1.1s
 => importing cache manifest from ghcr.io/ansible/awx_devel:devel                                                                                                                                         1.1s
 => [internal] load build context                                                                                                                                                                         0.0s
 => => transferring context: 47.82kB                                                                                                                                                                      0.0s
 => FROM quay.io/ansible/receptor:latest@sha256:f784c48779b9c23500b877b2941a55d5cbcbae345ea4277a053475560a296c6a                                                                                          8.9s
 => => resolve quay.io/ansible/receptor:latest@sha256:f784c48779b9c23500b877b2941a55d5cbcbae345ea4277a053475560a296c6a                                                                                    0.0s
 => => sha256:53773955bbbb686368a2a8a98ef51a69daaa29d577a3810edd9ea1df56463bd9 3.87kB / 3.87kB                                                                                                            0.2s
 => => sha256:f784c48779b9c23500b877b2941a55d5cbcbae345ea4277a053475560a296c6a 1.05kB / 1.05kB                                                                                                            0.0s
 => => sha256:655f569aa370a357aa9f95398a147c20738d87b41c56c3ff2225459ecde175e8 1.57kB / 1.57kB                                                                                                            0.0s
 => => sha256:aef9f04251dc09a0d985cb54e014b4601ab1fd78f3643d168a8e7b62dff8adec 4.35kB / 4.35kB                                                                                                            0.0s
 => => sha256:758fc7b8481884f402fc80a4e62a4c4cbcce62de7f89a54d9c612839b5390dba 58.02MB / 58.02MB                                                                                                          5.9s
 => => sha256:3b1cd45e21351c43c6a3aa61c3da80abfbcfd0e290c563935f2a7fa000b619fb 9.12kB / 9.12kB                                                                                                            0.3s
 => => sha256:7c50b0a3dbb1554ec1ab9fdbfd6ed92d83fc29ff4d8994adbe013f07d463cb39 230B / 230B                                                                                                                0.4s
 => => sha256:5a1174314f4615103fb6d7eceb7507e3b580c77d430f9f00e24245bc58f1d92f 7.87MB / 7.87MB                                                                                                            1.5s
 => => sha256:1c6b83b7d0e2c13993177d643928e3cd0052d6e590c6598ac1c53305056962b6 22.48MB / 22.48MB                                                                                                          3.5s
 => => extracting sha256:758fc7b8481884f402fc80a4e62a4c4cbcce62de7f89a54d9c612839b5390dba                                                                                                                 1.5s
 => => extracting sha256:3b1cd45e21351c43c6a3aa61c3da80abfbcfd0e290c563935f2a7fa000b619fb                                                                                                                 0.0s
 => => extracting sha256:53773955bbbb686368a2a8a98ef51a69daaa29d577a3810edd9ea1df56463bd9                                                                                                                 0.0s
 => => extracting sha256:7c50b0a3dbb1554ec1ab9fdbfd6ed92d83fc29ff4d8994adbe013f07d463cb39                                                                                                                 0.0s
 => => extracting sha256:5a1174314f4615103fb6d7eceb7507e3b580c77d430f9f00e24245bc58f1d92f                                                                                                                 0.5s
 => => extracting sha256:1c6b83b7d0e2c13993177d643928e3cd0052d6e590c6598ac1c53305056962b6                                                                                                                 0.5s
 => https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf                                                                                          0.1s
 => https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf                                                                                                 0.1s
 => [builder 1/9] FROM quay.io/centos/centos:stream9@sha256:0c192d18e6c8fa2515351757354c939d968d1f18bde3ca3a5ccc1d2d00038ff7                                                                              9.7s
 => => resolve quay.io/centos/centos:stream9@sha256:0c192d18e6c8fa2515351757354c939d968d1f18bde3ca3a5ccc1d2d00038ff7                                                                                      0.0s
 => => sha256:0c192d18e6c8fa2515351757354c939d968d1f18bde3ca3a5ccc1d2d00038ff7 858B / 858B                                                                                                                0.0s
 => => sha256:45b5137697e01a7a7aaea1cd3f5d56262e37e60ef8b8c819371dd5988670e216 350B / 350B                                                                                                                0.0s
 => => sha256:11aff08538f27c5adc21a036055d1a3e6b0b2ac09950455c06125faf80f8426a 1.16kB / 1.16kB                                                                                                            0.0s
 => => sha256:38fb86e1a5f8ed159b5f78103af148dac5e6c069ea50b548a2c29d4af710171f 57.90MB / 57.90MB                                                                                                          7.3s
 => => extracting sha256:38fb86e1a5f8ed159b5f78103af148dac5e6c069ea50b548a2c29d4af710171f                                                                                                                 2.2s
 => [builder 2/9] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' &&     dnf config-manager --set-enabled crb &&     dnf -y install     gcc     gcc-c++     git-core     gettext       83.0s
 => [builder 3/9] RUN pip3 install virtualenv build                                                                                                                                                       3.5s
 => [builder 4/9] ADD Makefile /tmp/Makefile                                                                                                                                                              0.1s
 => [builder 5/9] RUN mkdir /tmp/requirements                                                                                                                                                             0.3s
 => [builder 6/9] ADD requirements/requirements.txt     requirements/requirements_tower_uninstall.txt     requirements/requirements_git.txt     /tmp/requirements/                                        0.0s
 => [builder 7/9] RUN cd /tmp && make requirements_awx                                                                                                                                                  128.0s
 => [builder 8/9] ADD requirements/requirements_dev.txt /tmp/requirements                                                                                                                                 0.0s
 => [builder 9/9] RUN cd /tmp && make requirements_awx_dev                                                                                                                                               45.9s
 => CACHED [stage-1  2/32] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' &&     dnf config-manager --set-enabled crb &&     dnf -y install acl     git-core     git-lfs     glibc-la  0.0s
 => CACHED [stage-1  3/32] RUN pip3 install virtualenv supervisor dumb-init                                                                                                                               0.0s
 => CACHED [stage-1  4/32] RUN rm -rf /root/.cache && rm -rf /tmp/*                                                                                                                                       0.0s
 => CACHED [stage-1  5/32] RUN dnf -y install     crun     gdb     gtk3     gettext     hostname     procps     alsa-lib     libX11-xcb     libXScrnSaver     iproute     strace     vim     nmap-ncat    0.0s
 => CACHED [stage-1  6/32] RUN pip3 install black git+https://github.com/coderanger/supervisor-stdout setuptools-scm                                                                                      0.0s
 => CACHED [stage-1  7/32] RUN dnf --enablerepo=baseos-debug -y install python3-debuginfo || :                                                                                                            0.0s
 => CACHED [stage-1  8/32] RUN dnf install -y epel-next-release && dnf install -y inotify-tools && dnf remove -y epel-next-release                                                                       27.7s
 => => pulling sha256:a8a182e1208f4efce02936a724dfda723422ad9f6cabc918890377afbc8df2c8                                                                                                                    5.2s
 => => pulling sha256:b831cc6670b6c46936bd03c5da9d4901c7e7df52aa2c23ddf21850cecef15b36                                                                                                                    2.7s
 => => pulling sha256:7b860301d64d2e7a8eda078b663db67128e377565120bf67cccd7b7bf5e67338                                                                                                                    0.9s
 => => pulling sha256:1c658a296276f27f3782c734a5308d9593ef77752a417047a604497941ca65db                                                                                                                   19.3s
 => => pulling sha256:d88a6c6dffff7b0c3d3e586f4b8eec6a6bb07d2514a7392f807b905b479a214f                                                                                                                    0.9s
 => => pulling sha256:43e89b9ea82a8277e84b71637c39e77c9c54d65f8c6195031703e7a08611338e                                                                                                                    0.3s
 => => pulling sha256:5380235134a77ba4e8b8b21ccc7d5e5f3ac9691fecbb445f81bf0b66adb619f7                                                                                                                    4.3s
 => [stage-1  9/32] COPY --from=builder /var/lib/awx /var/lib/awx                                                                                                                                         4.1s
 => [stage-1 10/32] RUN ln -s /var/lib/awx/venv/awx/bin/awx-manage /usr/bin/awx-manage                                                                                                                    0.3s
 => [stage-1 11/32] COPY --from=quay.io/ansible/receptor:latest /usr/bin/receptor /usr/bin/receptor                                                                                                       0.3s
 => [stage-1 12/32] RUN openssl req -nodes -newkey rsa:2048 -keyout /etc/nginx/nginx.key -out /etc/nginx/nginx.csr         -subj "/C=US/ST=North Carolina/L=Durham/O=Ansible/OU=AWX Development/CN=awx.l  0.6s
 => [stage-1 13/32] RUN dnf install -y podman && rpm --restore shadow-utils 2>/dev/null                                                                                                                  13.4s
 => [stage-1 14/32] RUN sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage  0.4s
 => [stage-1 15/32] RUN mkdir -p /etc/containers/registries.conf.d/ && echo "unqualified-search-registries = []" >> /etc/containers/registries.conf.d/force-fully-qualified-images.conf && chmod 644 /et  0.4s
 => [stage-1 16/32] ADD tools/ansible/roles/dockerfile/files/rsyslog.conf /var/lib/awx/rsyslog/rsyslog.conf                                                                                               0.0s
 => [stage-1 17/32] ADD tools/ansible/roles/dockerfile/files/wait-for-migrations /usr/local/bin/wait-for-migrations                                                                                       0.0s
 => [stage-1 18/32] ADD tools/ansible/roles/dockerfile/files/stop-supervisor /usr/local/bin/stop-supervisor                                                                                               0.0s
 => [stage-1 19/32] ADD tools/docker-compose/launch_awx.sh /usr/bin/launch_awx.sh                                                                                                                         0.0s
 => [stage-1 20/32] ADD tools/docker-compose/nginx.conf /etc/nginx/nginx.conf                                                                                                                             0.0s
 => [stage-1 21/32] ADD tools/docker-compose/nginx.vh.default.conf /etc/nginx/conf.d/nginx.vh.default.conf                                                                                                0.0s
 => [stage-1 22/32] ADD tools/docker-compose/start_tests.sh /start_tests.sh                                                                                                                               0.0s
 => [stage-1 23/32] ADD tools/docker-compose/bootstrap_development.sh /usr/bin/bootstrap_development.sh                                                                                                   0.0s
 => [stage-1 24/32] ADD tools/docker-compose/entrypoint.sh /entrypoint.sh                                                                                                                                 0.0s
 => [stage-1 25/32] ADD tools/scripts/config-watcher /usr/bin/config-watcher                                                                                                                              0.0s
 => [stage-1 26/32] ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/containers.conf /etc/containers/containers.conf                                             0.0s
 => [stage-1 27/32] ADD https://raw.githubusercontent.com/containers/libpod/master/contrib/podmanimage/stable/podman-containers.conf /var/lib/awx/.config/containers/containers.conf                      0.0s
 => [stage-1 28/32] ADD tools/docker-compose/awx.egg-link /tmp/awx.egg-link                                                                                                                               0.0s
 => [stage-1 29/32] ADD tools/docker-compose/awx-manage /usr/local/bin/awx-manage                                                                                                                         0.0s
 => [stage-1 30/32] ADD tools/scripts/awx-python /usr/bin/awx-python                                                                                                                                      0.0s
 => [stage-1 31/32] RUN for dir in       /var/lib/awx       /var/lib/awx/rsyslog       /var/lib/awx/rsyslog/conf.d       /var/lib/awx/.local/share/containers/storage       /var/run/awx-rsyslog       /  0.3s
 => [stage-1 32/32] RUN for dir in       /etc/containers       /var/lib/awx/.config/containers       /var/lib/awx/.config/cni       /var/lib/awx/.local       /var/lib/awx/venv       /var/lib/awx/venv/  0.6s
 => preparing layers for inline cache                                                                                                                                                                     0.1s
ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF
make: *** [Makefile:530: docker-compose-build] Error 1

[Commifreak]

After searching a bit more - it seems to be an docker issue. Can someone confirm that (Ubuntu 22.04 with latest docker)?

[gm0neyl0ve]

I get the same issue. Ubuntu 22.04 Docker version 23.0.0, build e92dd87 @Commifreak is there a Docker issue you are tracking?

[Commifreak]

No. I have to lookup the link again. But there they talk about docker downgrade to male it work again.

[Commifreak]

Ahh, got it: https://github.com/moby/moby/issues/44918

[gonzalezr82]

Hello, I have a similar error when executing make docker-compose

When I try to run make docker-compose to recreate the containers from an AWX installation with docker it gives me the following error

ERROR: [root@awxdev awx-21.6.0]# make docker-compose make: python3.9: Command not found /bin/sh: python3.9: command not found ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/sources.yml \ -e awx_image=ghcr.io/ansible/awx_devel \ -e awx_image_tag=HEAD \ -e receptor_image=quay.io/ansible/receptor:devel \ -e control_plane_node_count=1 \ -e execution_node_count=2 \ -e minikube_container_group=false \ -e enable_keycloak=false \ -e enable_ldap=false \ -e enable_splunk=false \ -e enable_prometheus=false \ -e enable_grafana=false

PLAY [Render AWX Dockerfile and sources] ***

TASK [Gathering Facts] ***** ok: [localhost]

TASK [Create _sources directories] ***** ok: [localhost] => (item=secrets) ok: [localhost] => (item=receptor)

TASK [sources : Detect secrets] **** skipping: [localhost] => (item=pg_password) skipping: [localhost] => (item=secret_key) ok: [localhost] => (item=broadcast_websocket_secret) skipping: [localhost] => (item=admin_password)

TASK [sources : Generate secrets if needed] **** skipping: [localhost] => (item=pg_password) skipping: [localhost] => (item=secret_key) skipping: [localhost] => (item=broadcast_websocket_secret) skipping: [localhost] => (item=admin_password)

TASK [sources : Include generated secrets unless they are explicitly passed in] **** skipping: [localhost] => (item=None) skipping: [localhost] => (item=None) ok: [localhost] => (item=None) skipping: [localhost] => (item=None) ok: [localhost]

TASK [sources : Write out SECRET_KEY] ** ok: [localhost]

TASK [sources : Render configuration templates] **** ok: [localhost] => (item=database.py) ok: [localhost] => (item=websocket_secret.py) ok: [localhost] => (item=haproxy.cfg)

TASK [sources : Delete old local_settings.py] ** ok: [localhost]

TASK [sources : Copy local_settings.py] **** ok: [localhost]

TASK [sources : Get OS info for sdb] *** ok: [localhost]

TASK [sources : Get user UID] ** ok: [localhost]

TASK [sources : Set fact with user UID] **** ok: [localhost]

TASK [sources : Set global version if not provided] **** skipping: [localhost]

TASK [sources : Generate Private RSA key for signing work] ***** ok: [localhost]

TASK [sources : Generate public RSA key for signing work] ** ok: [localhost]

TASK [sources : Include LDAP tasks if enabled] ***** skipping: [localhost]

TASK [sources : Render Docker-Compose] ***** ok: [localhost]

TASK [sources : Render Receptor Config(s) for Control Plane] *** ok: [localhost] => (item=1)

TASK [sources : Render Receptor Hop Config] **** ok: [localhost]

TASK [sources : Render Receptor Worker Config(s)] ** ok: [localhost] => (item=1) ok: [localhost] => (item=2)

TASK [sources : Render prometheus config] ** skipping: [localhost]

PLAY RECAP ***** localhost : ok=17 changed=0 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0

docker-compose -f tools/docker-compose/_sources/docker-compose.yml up --remove-orphans make: docker-compose: Command not found make: *** [Makefile:483: docker-compose] Error 127

can someone help me with this error

[gm0neyl0ve]

@gonzalezr82 gonzalezr82
Your issue seems not at all the same. Do you have docker-compose installed? which docker-compose

[gm0neyl0ve]

I just tested this out after a fresh git pull and it seems resolved for me. @Commifreak

[shanemcd]

I'm going to close this since it's not a problem with AWX itself. FWIW, when I see this kind of thing I typically delete /var/lib/docker (or wherever the data lives on your system) and restart Docker and it does the trick.