Closed AleksandrKls closed 2 years ago
One more question. Why created so many connections?
Seeing the same on the latest devel:
@dmatthewsbnd251 what OS are you using?
@ryanpetrello Hello.Sorry to bother you, do you have any idea why this might be?
Nope - following the 17.1.0 instructions, I'm not seeing this on a fresh install. I haven't tried on an Ubuntu machine, though (however, I'm not sure why that should matter).
I also installed on OS Debian 10 - the result is the same
Centos 7.9 here
Sent from my iPhone
On Mar 16, 2021, at 10:30 AM, AlexanderKls @.***> wrote:
I also installed on OS Debian 10 - the result is the same
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
Same here happened on both Ubuntu 20.04 and CentOS 8
Same for me on latest devel 18.0.0 with the given install instructions. (clean install)
Ubuntu 18.04.5 LTS
Same on CentOS 8 Steam
@ryanpetrello It looks like a massive problem
Any update?
Same problem here. Debian 10. Migration from AWX 16 docker.
Hi! Same problem (also with latest awx release). Environment:
Not sure why this error only happens on some distros.
Can someone try modifying the image to create an nginx
user and see if that helps?
Not sure why this error only happens on some distros.
Can someone try modifying the image to create an
nginx
user and see if that helps?
Yes I did. It worked. But I have other issues...
We are also trying to get the whole thing up and running in an Amazon Linux 2. What I realised is the 3rd line of the awx_1 container as given in issue https://github.com/ansible/awx/issues/9866 which has been marked as dupe:
tools_awx_1 | Error: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
I checked the host file system (which is xfs) with the correct options for overlayfs being turned on:
xfs_info /
meta-data=/dev/nvme0n1p1 isize=512 agcount=26, agsize=524159 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0
data = bsize=4096 blocks=13106683, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Docker is also configured properly (default settings):
]# docker info
...
Server Version: 19.03.13-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
...
We made some progress. I also tried to modify the Makefile for version 19.0.0 - i.e. the section nginx:
nginx:
useradd -g nginx nginx
nginx -g "daemon off;"
That way supervisord
will - with every start of nginx - add the user. You obviously get an error useradd: user 'nginx' already exists
if the user is already there - but for us the services started properly.
That worked! Very nice.
If it's not obvious please note you need to do make docker-compose-build
after following this solution:
We made some progress. I also tried to modify the Makefile for version 19.0.0 - i.e. the section
nginx:
nginx: useradd -g nginx nginx nginx -g "daemon off;"
That way
supervisord
will - with every start of nginx - add the user. You obviously get an erroruseradd: user 'nginx' already exists
if the user is already there - but for us the services started properly.
We are also trying to get the whole thing up and running in an Amazon Linux 2. What I realised is the 3rd line of the awx_1 container as given in issue #9866 which has been marked as dupe:
tools_awx_1 | Error: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
I checked the host file system (which is xfs) with the correct options for overlayfs being turned on:
xfs_info / meta-data=/dev/nvme0n1p1 isize=512 agcount=26, agsize=524159 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1 spinodes=0 data = bsize=4096 blocks=13106683, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
Docker is also configured properly (default settings):
]# docker info ... Server Version: 19.03.13-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true ...
Yes, I fixed the nginx error but now, jobs can't running :
WARN[0000] Found deprecated file /etc/containers/libpod.conf, please remove. Use /etc/containers/containers.conf to override defaults.
WARN[0000] Found deprecated file /etc/containers/libpod.conf, please remove. Use /etc/containers/containers.conf to override defaults.
Error: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
I fixed issue with modify /etc/containers/storage.conf
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"
Now new error
ERRO[0032] Error adding network: running [/usr/sbin/iptables -t nat -A POSTROUTING -s 10.88.0.2 -j CNI-021a81a5f0e6e4b17ee32e18 -m comment --comment name: "podman" id: "2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72" --wait]: exit status 4: iptables v1.8.4 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain POSTROUTING
ERRO[0032] Error while adding pod to CNI network "podman": running [/usr/sbin/iptables -t nat -A POSTROUTING -s 10.88.0.2 -j CNI-021a81a5f0e6e4b17ee32e18 -m comment --comment name: "podman" id: "2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72" --wait]: exit status 4: iptables v1.8.4 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain POSTROUTING
ERRO[0032] Error preparing container 2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72: error configuring network namespace for container 2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72: running [/usr/sbin/iptables -t nat -A POSTROUTING -s 10.88.0.2 -j CNI-021a81a5f0e6e4b17ee32e18 -m comment --comment name: "podman" id: "2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72" --wait]: exit status 4: iptables v1.8.4 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain POSTROUTING
Error: error resolving storage path for container 2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72: lstat /var/lib/containers/storage/overlay/0f8ff6edf67258601ab5eb20fe137d358516fb2d20653598319ce06d9822aa0e/merged: invalid argument
I tried to reproduce this on Debian 10 and was unable to. Here is my docker info
, in case it helps anyone:
$ docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.7.0)
Server:
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 3
Server Version: 20.10.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.19.0-16-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.832GiB
Name: debian
ID: VIAV:IXWF:EI7W:I2HS:ZKIX:5SCX:TPN3:RAMS:FMPS:D5NE:LA4E:LHBR
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
@shanemcd I think the issue not come from docker but podman in the tools_awx_1 container.
AWX works but when I try to run a job, awx fail to run a podman container in tools_awx_1.
Issue is about network and iptables.
I can confirm issue with awx image because I can run podman with other container :
docker run --privileged -v /tmp/podman:/var/lib/containers marshallford/podman:latest run hello-world
Same issue with release 19.1.0 on Debian 10. Ngnix user does not exist and AWX cannot be used. The Makefile work-a-round fixes the issue.
useradd -g nginx nginx
nginx -g "daemon off;"```
For those who want to automate this step.
`sed -i '/^nginx:/a \\tuseradd -g nginx nginx' Makefile`
I've been through the more or less the same steps as ktibi and got the same results.
I fixed issue with modify /etc/containers/storage.conf
[storage.options] mount_program = "/usr/bin/fuse-overlayfs"
Now new error
ERRO[0032] Error adding network: running [/usr/sbin/iptables -t nat -A POSTROUTING -s 10.88.0.2 -j CNI-021a81a5f0e6e4b17ee32e18 -m comment --comment name: "podman" id: "2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72" --wait]: exit status 4: iptables v1.8.4 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain POSTROUTING ERRO[0032] Error while adding pod to CNI network "podman": running [/usr/sbin/iptables -t nat -A POSTROUTING -s 10.88.0.2 -j CNI-021a81a5f0e6e4b17ee32e18 -m comment --comment name: "podman" id: "2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72" --wait]: exit status 4: iptables v1.8.4 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain POSTROUTING ERRO[0032] Error preparing container 2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72: error configuring network namespace for container 2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72: running [/usr/sbin/iptables -t nat -A POSTROUTING -s 10.88.0.2 -j CNI-021a81a5f0e6e4b17ee32e18 -m comment --comment name: "podman" id: "2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72" --wait]: exit status 4: iptables v1.8.4 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain POSTROUTING Error: error resolving storage path for container 2a1be3ecbdd64e66f19236d5758dda76174c6e214ee57b71c645e9a924d7ac72: lstat /var/lib/containers/storage/overlay/0f8ff6edf67258601ab5eb20fe137d358516fb2d20653598319ce06d9822aa0e/merged: invalid argument
Did you find some workaround for that?
We made some progress. I also tried to modify the Makefile for version 19.0.0 - i.e. the section
nginx:
nginx: useradd -g nginx nginx nginx -g "daemon off;"
That way
supervisord
will - with every start of nginx - add the user. You obviously get an erroruseradd: user 'nginx' already exists
if the user is already there - but for us the services started properly.
This works for me, but i'm curious if there are any ramifications to doing this. How is this supposed to work? Or was this fix just something that was missed?
We made some progress. I also tried to modify the Makefile for version 19.0.0 - i.e. the section
nginx:
nginx: useradd -g nginx nginx nginx -g "daemon off;"
That way
supervisord
will - with every start of nginx - add the user. You obviously get an erroruseradd: user 'nginx' already exists
if the user is already there - but for us the services started properly.
If you make it -useradd -g nginx nginx
you might avoid the warning/error, too.
ISSUE TYPE
COMPONENT NAME
ENVIRONMENT
STEPS TO REPRODUCE
Installation according to instructions
Var broadcast_websocket_secret - not used.
EXPECTED RESULTS
Stage: Run AWX Start command "make docker-compose"
Nginx failed
awx_1_1 | 2021-03-11 00:08:31,423 WARNING [-] awx.main.commands.run_callback_receiver scaling up worker pid:339 awx_1_1 | 2021-03-11 00:08:31,425 INFO success: awx-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) awx_1_1 | 2021-03-11 00:08:31,427 INFO spawned: 'awx-nginx' with pid 340 awx_1_1 | tower-processes:awx-dispatcher: stopped awx_1_1 | tower-processes:awx-receiver: stopped awx_1_1 | tower-processes:awx-dispatcher: started awx_1_1 | tower-processes:awx-receiver: started awx_1_1 | 2021-03-11 00:08:31,431 WARNING [-] awx.main.commands.run_callback_receiver scaling up worker pid:341 awx_1_1 | make[1]: Entering directory '/awx_devel' awx_1_1 | 2021-03-11 00:08:31,437 WARNING [-] awx.main.commands.run_callback_receiver scaling up worker pid:343 awx_1_1 | 2021-03-11 00:08:31,443 WARNING [-] awx.main.commands.run_callback_receiver scaling up worker pid:346 awx_1_1 | nginx -g "daemon off;" awx_1_1 | nginx: [emerg] getpwnam("nginx") failed awx_1_1 | make[1]: *** [Makefile:263: nginx] Error 1 awx_1_1 | make[1]: Leaving directory '/awx_devel' awx_1_1 | 2021-03-11 00:08:31,455 INFO exited: awx-nginx (exit status 2; not expected) awx_1_1 | 2021-03-11 00:08:31,455 INFO gave up: awx-nginx entered FATAL state, too many start retries too quickly awx_1_1 | 2021-03-11 00:08:32,457 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
And console is intercepted
redis_1_1 | 1:M 11 Mar 2021 00:13:12.026 100 changes in 300 seconds. Saving... redis_1_1 | 1:M 11 Mar 2021 00:13:12.026 Background saving started by pid 10 redis_1_1 | 10:C 11 Mar 2021 00:13:12.039 DB saved on disk redis_1_1 | 10:C 11 Mar 2021 00:13:12.039 RDB: 0 MB of memory used by copy-on-write redis_1_1 | 1:M 11 Mar 2021 00:13:12.126 Background saving terminated with success redis_1_1 | 1:M 11 Mar 2021 00:18:13.084 100 changes in 300 seconds. Saving...
But containers started: docker ps
9dcc24b88221 gcr.io/ansible-tower-engineering/awx_devel:devel "/entrypoint.sh laun…" 51 minutes ago Up 51 minutes 0.0.0.0:6899->6899/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 0.0.0.0:8080->8080/tcp, 22/tcp, 0.0.0.0:8888->8888/tcp tools_awx_1
9f4772187d15 postgres:12 "docker-entrypoint.s…" 51 minutes ago Up 51 minutes 5432/tcp tools_postgres_1
f26eac102cf4 redis:latest "redis-server /usr/l…" 51 minutes ago Up 51 minutes 6379/tcp tools_redis_1
ACTUAL RESULTS
Сommand should complete successfully and start the web server
ADDITIONAL INFORMATION
The virtual machine is created specifically for AWX. On this host there is nothing but AWX and mariadb.