Portal container crashes on RHEL8 Hypver-V VM using podman-compose and podman

[MaximUltimatum]

The Issue I'm facing, and how to reproduce it

I'm attempting to work my way through the lab using podman and podman-compose. This has been going acceptably so far - I've needed to make some tweaks to how the containers are networked, and run podman-compose with sudo. Currently, when I run sudo podman-compose -f docker-compose.yaml -t identity up I get the following error after the portal container starts up, and if I then check the container status with sudo podman ps -a it shows as exited. All the other containers will still be running.

2021/10/22 12:16:14 [emerg] 1#1: host not found in upstream "centos1" in /etc/nginx/conf.d/default.conf:15
nginx: [emerg] host not found in upstream "centos1" in /etc/nginx/conf.d/default.conf:15

Things I've Tried

Downgrading to cgroups1 with sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" And then killing all containers, then sudo podman rm --all, sudo podman-compose pull, and starting everything up again.

Possibly Helpful Information

The contents of my config file:

# sshd ports
UBUNTUC_PORT_SSHD=2221
UBUNTU1_PORT_SSHD=2222
UBUNTU2_PORT_SSHD=2223
UBUNTU3_PORT_SSHD=2224
CENTOS1_PORT_SSHD=2225
CENTOS2_PORT_SSHD=2226
CENTOS3_PORT_SSHD=2227

# ttyd (web terminal) ports
UBUNTUC_PORT_TTYD=7681
UBUNTU1_PORT_TTYD=7682
UBUNTU2_PORT_TTYD=7683
UBUNTU3_PORT_TTYD=7684
CENTOS1_PORT_TTYD=7685
CENTOS2_PORT_TTYD=7686
CENTOS3_PORT_TTYD=7687

# Shared config volume
CONFIG=/home/admin/git/diveintoansible-lab/config

# Shared home directories
ANSIBLE_HOME=/home/admin/git/diveintoansible-lab/ansible_home

Output of cat /proc/cmdline

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-305.12.1.el8_4.x86_64 root=/dev/mapper/rhel-root ro resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet video=hyperv_fb:1920x1080 systemd.unified_cgroup_hierarchy=0

Output of mount | grep cgroup

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=1962388k,nr_inodes=490597,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)

Output of cat /etc/default/grub

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet video=hyperv_fb:1920x1080 systemd.unified_cgroup_hierarchy=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

Output of grep cgroup /proc/filesystems

nodev   cgroup
nodev   cgroup2

[spurin]

Thanks @MaximUltimatum for raising this, can you please share the output of

ls /sys/fs/cgroup

[MaximUltimatum]

@spurin The output is

sudo ls /sys/fs/cgroup
blkio  cpu  cpuacct  cpu,cpuacct  cpuset  devices  freezer  hugetlb  memory  net_cls  net_cls,net_prio  net_prio  perf_event  pids  rdma  systemd

[spurin]

Looks like you're already running cgroups v1 based on that output. If you run the following, does it show any errors -

podman run --rm -v /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged spurin/diveintoansible:ansible

When this is working as expected, there will be no output (as systemd kicks in)

[spurin]

Also @MaximUltimatum it may be worthwhile trying the command above with --systemd=always

[MaximUltimatum]

@spurin There is indeed no output or errors (and terminal control is held). Going to a new terminal and running podman ps -a returns the following

CONTAINER ID  IMAGE                                     COMMAND               CREATED         STATUS             PORTS       NAMES
bdfa6a6c8638  docker.io/spurin/diveintoansible:ansible  /lib/systemd/syst...  49 seconds ago  Up 49 seconds ago              silly_ride

[spurin]

Okay so that looks promising. CTRL-C and then please add the following to the command -

-p 2222:22 -p 7681:7681

Then, let's see if you can connect via http on http://localhost:7681 and/or ssh to it on localhost:2222

[spurin]

(By the way, at this point as there is no volume mount you should be able to use guest/guest or root/root, if you get a prompt)

[MaximUltimatum]

I'm able to login with guest/guest from the browser from the redhat desktop!

[spurin]

Okay great, good progress so far. Can you please add the following to the command line, adjusting accordingly to your paths.

-v /home/admin/git/diveintoansible-lab/config:/config -v /home/admin/git/diveintoansible-lab/ansible_home/ubuntu-c/ansible:/home/ansible

Don't worry about the ansible_home subdirectories existing, these should get created on startup

Update - I see you shared the .env file above, have updated the command accordingly

[MaximUltimatum]

That completed successfully, I am able to log in with ansible/password now

[spurin]

Okay great. That works for ubuntu ones, let's toggle the command and check if a centos one works. Can you please change the image to -

spurin/diveintoansible:centos

On your command, and try launching (logging in etc)

[MaximUltimatum]

That also works. I'm able to log into the machine

[spurin]

Hi @MaximUltimatum

I also downloaded and have setup a RHEL 8.4 instance. I'm seeing the same issue and it seems to be specific with the portal image. Within this, I configure reverse proxies to the other systems and therefore, it needs to have connectivity by hostname to those other containers for this to work.

I've run out of time today but, will continue to look at this further. If you wish to troubleshoot some more, you could try changing the portal image to -

nginx:1.19

I use this as a base for spurin/diveintoansible:portal and during the build, customisations are made. I think it's worth changing to this image, then, getting it up and running, executing a shell to the container and then seeing if networking is working as expected to the other hosts.

When I'm back next, I'll continue looking into it also 👍

[MaximUltimatum]

Sounds good. Many thanks for your help!

[MaximUltimatum]

Hey @spurin. Didn't get a whole lot of time to plug away at this today, but I did try swapping in that nginx base image into the docker-compose.yml file. I exec'd into the terminal and added ping and nslookup to the nginx image and then attempted to do nslookup and ping on ubuntu1 and ubuntu-c, neither of which worked. I think this either means that the networking isn't configured correctly in the compose or I'm missing something else

[spurin]

Thanks @MaximUltimatum, docker-compose is helping with the networking and the simplicity was one of the reason I used it. By using that network directive in the compose file, you get dns resolution and container port mapping as standard without any manual user networking.

Looking into this more, it looks as if podman-compose isn't quite there on it being a native 1-2-1 alternative but, there may still be some ways in which we can get this working.

The following comment from 17 days ago looks promising (the whole thread is interesting too) -

https://github.com/containers/podman-compose/issues/288#issuecomment-935901679

If you get a spare moment have a look otherwise I'll do so when I'm back 👍

[spurin]

Hi @MaximUltimatum

Just checking in and letting you know that this is still on my radar. I've been working on newer images that work both with cgroups v1 and v2. They are now available for testing as per the release-candidate branch and I think that for these efforts, we're likely to have more success with this approach.

If you do give them a try, you should be able to remove the /sys/fs/cgroup:/sys/fs/cgroup:ro volume mount.

Best Regards

James

[MaximUltimatum]

@spurin Thank you! I apologize, I haven't had time to circle back around to this yet. I will give the release candidates a try this weekend!

[spurin]

Hi @MaximUltimatum

Have been playing around with this a bit further. The release-candidate images work well with steps we've outlined above. If it's run in rootfull mode using network_mode as mentioned in the comment above (https://github.com/spurin/diveintoansible-lab/issues/68#issuecomment-949997073) allows each container to have it's own ip address.

These however, seem to change upon every iteration and there doesn't seem to be any name resolution (which is why the portal fails on startup).

Open to options on 'resolving' this (pun intended). I tried using avahi on some of the containers and it worked ... they were resolvable as ubuntu-c.local or ubuntu1.local, but ... the convenience of not having ubuntu-c is a bit inconvenient, especially as every example would need to be updated. Just to note, a search keyword in /etc/resolv.conf for local doesn't work as avahi doesn't support this.

Will continue to tinker with this in the background. Let me know if you have any further progression or ideas.

[spurin]

@MaximUltimatum

Further to my post above I'm happy to report that I've got this working. As well as the above that I mentioned with rootfull networking and network_mode, the custom network needed to make use of the 'dnsname' plugin (https://github.com/containers/dnsname). This had to be compiled and copied but once it was in place (it's just a single binary), it worked as expected.

A screenshot -

I'll clean up the steps/process to get it working and will follow-up on this thread.

Thanks

James

[spurin]

@MaximUltimatum please see - https://github.com/spurin/diveintoansible-lab/blob/podman-compose/README.md

Let me know how your testing goes 😊

[MaximUltimatum]

@spurin - Thank you so much for working on this! I have done some testing on the podman-compose branch. Upon running sudo podman-compose -t identity -f podman-compose.yaml up The containers error out with Error: statfs /home/admin/diveintoansible-lab/ansible_home/shared: no such file or directory I've reviewed the earlier videos in your ansible course to see if these are manually created, but as best I can tell the mounted ansible_home folder was created by the container setup. Am I missing anything obvious?

Things I've tried - disabling SELinux

I have copied the dnsname precompiled binary to /usr/local/libexec/cni/ and created the network.

One (possibly unrelated) issue - the dnsname plugin is not appearing in the list of network plugins -

NETWORK ID    NAME                         VERSION     PLUGINS
2f259bab93aa  podman                       0.4.0       bridge,portmap,firewall,tuning
864bd55e019e  diveinto.io                  0.4.0       bridge,portmap,firewall,tuning
224d355b5a26  docker-practice_default      0.4.0       bridge,portmap,firewall,tuning

[spurin]

Thanks @MaximUltimatum

Could you please check out the repository again, have added the shared folder (git ignores empty folders, so this has a hidden .keep file).

I've updated the README file with some additional instructions, specifically around the user home directories, and what to do after the CNI driver is copied. I think this is the missing step (once done, all future networks get the dnsname plugin by default).

Best Regards

James

[MaximUltimatum]

@spurin - Looking great! I made a slight tweak that I suspect may be distro-dependent, I copied the the dnsname plugin to /usr/libexec/cni based off this message of valid locations [failed to find plugin "dnsname" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]

After that modification, all networking appears to be functioning. I can access the web console, ssh to any of the child ubuntu/centos vms, and use the web console :D

I've really appreciated your help with this. Let me know if you ever need a beta tester for something or a second pair of hands - I owe you one!

[spurin]

@MaximUltimatum

There's no debt owed at all, I'm very happy that you pursued this venture and it's a been a great learning journey working through this with you. Thanks!

Regarding the '/usr/libexec/cni' location, I've just updated the docs. This was my bad, the instructions for dnsname actually state to use the previous location and I also encountered the same issue. As I was recalling the process I made reference to the first attempt.

In case you didn't see, you're thanked in the credits for this on the repo (in the readme). Please connect on LinkedIn and if you make any further enhancements on this, message me or ping me a merge request.

Best Regards

James Spurin