What are considerations when migrating to Podman?

[joshuacurtiss]

Given that our new VMs are Red Hat 9, and RHEL 8 and 9 do not support Docker, we must learn how to use Podman. Whereas it is marketed as practically a drop-in replacement for Docker, the reality is that there are challenges and hiccups along them way.

This issue can track all the lessons learned while migrating from Docker to Podman as we implement our container environment for RHEL 9.

[joshuacurtiss]

PROBLEM: Compatibility with Docker Compose files.

Generally, the answer is to use podman-compose, a project that simulates the behavior of docker-compose.

Just as docker-compose has been deprecated for use of the built-in and now better maintained docker compose subcommand, so too podman has been enhanced to use a podman compose command that will just look for the nearest compose spec handler, whether it's podman-compose or docker-compose, which carries on the similar behavior of having a compose subcommand. However, this feature was not yet provided in the version of podman natively provided in RHEL-9, podman 4.6.1. What this means is we need to explicitly use podman-compose for now.

One way to handle compatibility is to make all docker commands be callable as docker commands but they link to podman commands when on a podman system. Part of the podman install for our RHEL-9 setup is to install the container-tools package which includes podman-docker which provides those links. But it doesn't handle docker-compose/podman-compose. So, we have our own docker-compose script that is added to the path so that docker-compose calls just reference podman-compose.

That said, podman-compose behaviors a little different than docker-compose and those nuances can be frustrating. For instance:

1. podman-compose may not readily see your script variables since it runs as a separate script, so be sure to export your variables that you want to be substituted in the compose definitions.
2. podman-compose may differ from docker-compose in what it thinks the working directory is. To be safe, always start a subshell and cd into the directory you want to be the working directory before you execute your docker-compose/podman-compose command. (i.e. (cd /my/path && docker-compose ...))
3. podman-compose may differ from docker-compose in how they name your containers, for instance, whether to use underscores or dashes. Don't hard-code expected container names unless you've explicitly defined the name in the compose file.

4. In a semi-related vein, you may need your podman implementation to use pods (see the question on services and container persistence on reboots), but you don't want to introduce podman-specific code that will break in a docker environment. So, use logic to check whether you're in a podman environment or not, and add those values separately. For instance:

   # Good way to make a boolean that knows if your're using podman
   [[ $(docker --version) == podman* ]] && IS_PODMAN=true || IS_PODMAN=false
   
   # Starts the stack, but uses a pod if in podman
   $IS_PODMAN && ! docker pod exists "$pod_name" && podman pod create --name "$pod_name"
   $IS_PODMAN && podman_args=('--podman-run-args' "--pod $pod_name")
   (cd "$proj_dir" && docker-compose "${podman_args[@]}" -f "$docker_compose_path" up -d)
   
   # Stops a stack, removing the pod if in podman
   $IS_PODMAN && podman_args=('--podman-rm-args "-i"')
   (cd "$proj_dir" && docker-compose "${podman_args[@]}" -f "$docker_compose_path" down)
   $IS_PODMAN && docker pod exists "$pod_name" && podman pod rm -f "$pod_name"

[joshuacurtiss]

PROBLEM: Docker CLI compatibility.

This is answered in the docker compose question above. Basically, we install container-tools which includes a podman-docker package that helps provide links between docker and podman commands. So you can just use docker commands and it will use podman. This is not without its pitfalls, but it works and with experience you learn the pitfalls.

[joshuacurtiss]

PROBLEM: Multiple networks per container for container isolation, especially in reverse proxy scenarios.

At first glance, and after much bashing of your head into the wall, you'll think that podman can't support external networks, and that can be further confusing by talk of how the podman network layer is much different from podman's.

Thankfully, the reality is it does support multiple networks, but RHEL probably starts with extremely strict networking settings. Se the article Connecting multiple networks to a Podman container. The instructions are that easy and implemented in the RHEL-9 version of our docker installation script.

Basically, you just need:

sysctl -w net.ipv4.conf.all.rp_filter=2
echo "net.ipv4.conf.all.rp_filter=2" >> /etc/sysctl.conf

[joshuacurtiss]

PROBLEM: Privileged containers for publishing ports <1024 or other high-privilege requirements.

Generally, a benefit of podman is running containers in rootless mode, but docker runs as root, and it just means to be careful with how you use your containers.

Use privileged: true when a container needs high privileges.

[joshuacurtiss]

PROBLEM: Secrets operate different in Swarm, Docker Compose, and Podman.

Yes, Docker Swarm made implementing secrets very nice and easy. However, the Docker Compose spec still supports secrets that you just point to your own data files holding the secrets. Keep these at a path that is not viewable without escalated privileges.

For an example, look at the Nginx Proxy Manager setup. We generate long passwords and then reference the secrets in nginx-proxy-manager.yml.

[joshuacurtiss]

PROBLEM: Enabling Podman containers to persist on reboots.

In the past, podman suggested created a systemd service with the built-in podman generate systemd command. The newest version of podman has deprecated this approach in favor of Quadlets, but since our current version of Podman is 4.6.1 and the recommended approach is systemd, we use that.

If your project is a single container, you can just create a service for the container using the native podman generate systemd command and you're done. If you have multiple services (in other words, multiple containers), it's indisputably easier to embrace pods, because podman will properly create the services for all the containers under one master service that acts on the pod.

To help with the creation of services for either a container or a pod, I added podman-install-service which gets installed in the path. Once you bring up a pod, you can just run podman-install-service mypodname and it will create the services and enable them.

References:
https://www.reddit.com/r/podman/comments/171advd/whats_the_canonical_way_to_make_a_podman/
https://www.redhat.com/sysadmin/container-systemd-persist-reboot