Firewall Issue

[ReinerNippes]

Hi,

I run your role in a playbook on a CentOS8. All plain vanilla, nothing changed.

Pods are running. Logs are fine.

curl 127.0.0.1:8052 doesn't work. Time out. curl :8052 same result.

Any advice where to look? I'm a bit frustrated. Google "podman & firewalld" didn't really help.

Before I run your playbook I followed one of examples

podman run -dt -p 8080:8080/tcp -e HTTPD_VAR_RUN=/var/run/httpd -e HTTPD_MAIN_CONF_D_PATH=/etc/httpd/conf.d \
                  -e HTTPD_MAIN_CONF_PATH=/etc/httpd/conf \
                  -e HTTPD_CONTAINER_SCRIPTS_PATH=/usr/share/container-scripts/httpd/ \
                  registry.fedoraproject.org/f29/httpd /usr/bin/run-httpd

with worked fine. In terms of curl 127.0.0.1:8080

Thanks, Reiner

[cfelder]

Have you checked ss -tlpn for open ports?

[ReinerNippes]

[root@vdr01 ~]# ss -tlpn
State                  Recv-Q                 Send-Q                                  Local Address:Port                                   Peer Address:Port                                                                    
LISTEN                 4                      65535                                         0.0.0.0:8080                                        0.0.0.0:*                     users:(("conmon",pid=35488,fd=5))                 
LISTEN                 0                      128                                           0.0.0.0:22                                          0.0.0.0:*                     users:(("sshd",pid=33915,fd=4))                   

this is not the original awx role. i tried to run nextcloud. but at the end it's the same result. a running pod. not reachable.

[root@vdr01 ~]# podman ps
CONTAINER ID  IMAGE                                   COMMAND               CREATED         STATUS             PORTS                 NAMES
f954b56a1c5e  docker.io/library/nginx:alpine          /docker-entrypoin...  11 minutes ago  Up 11 minutes ago  0.0.0.0:8080->80/tcp  nextcloud_web
2cff8c94ede8  docker.io/library/nextcloud:fpm-alpine  /entrypoint.sh ph...  11 minutes ago  Up 11 minutes ago  0.0.0.0:8080->80/tcp  nextcloud_app
31238a991f86  docker.io/library/redis:alpine          docker-entrypoint...  11 minutes ago  Up 11 minutes ago  0.0.0.0:8080->80/tcp  nextcloud_redis
c2d0412ea9e0  docker.io/library/mariadb:10.5          docker-entrypoint...  11 minutes ago  Up 11 minutes ago  0.0.0.0:8080->80/tcp  nextcloud_mariadb

(sorry. shouldn't be so confusing. but i'm a total newby to podman and firewalld. trying to reuse my docker knowledge.)

[cfelder]

your podman ps output doesn't list any awx containers...

[ReinerNippes]

yes. this is not the original awx role. i modified the original playbook to run nextcloud. nevertheless i get the same result with the awx roles.

[cfelder]

nextcloud containers are fine and bound to host port 8080.

[ReinerNippes]

nope. curl 127.0.0.1:8080 -> time out.

[ReinerNippes]

on a fresh centos8.

[ec2-user@vdr01 ~]$ sudo -i
[root@vdr01 ~]# mkdir awx
[root@vdr01 ~]# cd awx
[root@vdr01 awx]# mkdir roles
[root@vdr01 awx]# vi my-awx.yml
[root@vdr01 awx]# cat >>roles/requirements.yml<<EOF
> ---
> - src: ikke_t.awx_pod
>   name: awx_pod
> - src: ikke_t.podman_container_systemd
>   name: podman_container_systemd
> EOF
[root@vdr01 awx]# 
[root@vdr01 awx]# yum install ansible -y
...
Installed:
  ansible-2.9.11-1.el8.noarch         libsodium-1.0.18-2.el8.x86_64   python3-bcrypt-3.1.6-2.el8.1.x86_64   python3-jmespath-0.9.0-11.el8.noarch   python3-paramiko-2.4.3-1.el8.noarch   python3-pyasn1-0.3.7-6.el8.noarch  
  python3-pynacl-1.3.0-5.el8.x86_64   sshpass-1.06-9.el8.x86_64      

Complete!
[root@vdr01 awx]# ansible-galaxy install --roles-path roles -r roles/requirements.yml
- downloading role 'awx_pod', owned by ikke_t
- downloading role from https://github.com/ikke-t/awx_pod/archive/10.0.0-0.tar.gz
- extracting awx_pod to /root/awx/roles/awx_pod
- awx_pod (10.0.0-0) was installed successfully
- downloading role 'podman_container_systemd', owned by ikke_t
- downloading role from https://github.com/ikke-t/podman-container-systemd/archive/master.tar.gz
- extracting podman_container_systemd to /root/awx/roles/podman_container_systemd
- podman_container_systemd (master) was installed successfully
- adding dependency: systemli.apt_repositories
- downloading role 'apt_repositories', owned by systemli
- downloading role from https://github.com/systemli/ansible-role-apt_repositories/archive/master.tar.gz
- extracting systemli.apt_repositories to /root/awx/roles/systemli.apt_repositories
- systemli.apt_repositories (master) was installed successfully
[root@vdr01 awx]# ansible-playbook -i localhost, my-awx.yml --connection=local
...
[root@vdr01 awx]# podman ps
CONTAINER ID  IMAGE                                   COMMAND               CREATED         STATUS             PORTS                   NAMES
a09bc187afaa  docker.io/library/redis:latest          docker-entrypoint...  21 seconds ago  Up 18 seconds ago  0.0.0.0:8052->8052/tcp  awx_redis
22485e34b04e  docker.io/ansible/awx_task:10.0.0       /tini -- /bin/sh ...  21 seconds ago  Up 18 seconds ago  0.0.0.0:8052->8052/tcp  awx_awxtask
d6fb21f8e887  docker.io/ansible/awx_web:10.0.0        /tini -- /bin/sh ...  21 seconds ago  Up 19 seconds ago  0.0.0.0:8052->8052/tcp  awx_awxweb
b4c239e268e7  docker.io/library/memcached:alpine      docker-entrypoint...  21 seconds ago  Up 19 seconds ago  0.0.0.0:8052->8052/tcp  awx_memcached
70fb4628600d  docker.io/centos/postgresql-10-centos7  container-entrypo...  21 seconds ago  Up 20 seconds ago  0.0.0.0:8052->8052/tcp  awx_postgres
[root@vdr01 awx]# curl 127.0.0.1:8052

^C

[root@vdr01 awx]# curl ifconfig.me 
195.201.5.81
[root@vdr01 awx]# 
[root@vdr01 awx]# curl 195.201.5.81:8052
^C

how to reach the pod?

[cfelder]

you may have to fix your cni configuration file /etc/cni/net.d/podman.conflist

please check the firewall backend does not contain "iptables"

just leave it empty

      {
         "type": "firewall",
         "backend": ""
      },

[ikke-t]

Hi,

yes, please check that. If you don't habe fresh install, the file has "iptables" there, which breaks things. I recall I changed it to firewalld and got things wirking again. See a related issue: https://github.com/containers/podman/issues/5316

BTW, I run my NextCloud like this: https://raw.githubusercontent.com/ikke-t/ansible-podman-examples/master/run-container-nextcloud-podman.yml

[ReinerNippes]

I have only a /etc/cni/net.d/87-podman-bridge.conflist with the following content:

{
  "cniVersion": "0.3.0",
  "name": "podman",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

I'm trying to port my playbooks https://github.com/ReinerNippes/nextcloud_on_docker or https://github.com/ReinerNippes/selfhosted_on_docker to CentOS8 and use podman instead of docker.

Any change to get something like this done?

[cfelder]

just another guess:

Are you running with selinux in enforced mode?

[ikke-t]

That's cool project! I have a suggestion, use cockpit web gui to control the podman & host. It could be used to tune anything within the host. I have a playbook to install that with Let's Encrypt certs. There's also role for DDNS:

https://github.com/RedHatNordicsSA/iot-hack/tree/master/roles/cockpit

What comes to networking, I'd suggest you create a small reproducal and an issue to podman. The team is very helpful. For that, I suggest you do simple pod creation and add containers into it from command line, so it's easy for them to reproduce. "podman pod create -p 8080:8080 nextcloud; podman run ..." and attach containers to it. This way you see if the problem is in this a single role or in the podman itself. If podman, create issue there.

[ReinerNippes]

[ec2-user@vdr01 ~]$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31

turning it off helps. but what would be the correct setting?

[ikke-t]

Correct setting is to have selinux enforcing. Take a look what selinux is blocking in your case. You could set it off for test time, and then capture the audit logs. It will list you what would have got blocked in case it was enforcing.

[ikke-t]

If you have Red Hat account, or you create developer account there, this article gives you guidance: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_selinux/troubleshooting-problems-related-to-selinux_using-selinux

[ReinerNippes]

thanks for your assistance. nevertheless I decided to drop podman support in my playbooks till there will be better ansible modules. right now the difference between a docker and a podman setup is to large for me.