Add support for running ELK with Podman

[geragio]

Hi,

It would be nice to add in the official documentation the possibility to run the Elastic stack using Podman Docker is not supported on RHEL 8/CentOS 8, so Podman is the only chance to run containers on these operating systems.

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

@pugnascotia I'm not up to speed on all the new hip alternative container runtimes. What's the deal with podman, how does it differ from Docker, can it just run normal OCI-compatible images, what would it take to support this?

[pugnascotia]

I haven't used podman, but this article is a decent overview. Podman supports OCI-compatible images, so as far as I understand it, there's no code or artifact changes to be made, it's just a question of whether we want to document using our images with it.

According to this article and the manpage, the podman CLI is intentionally compatible with the docker CLI. Would it be enough to call out in the docs that the commands should just work if you swap docker with podman?

[mark-vieira]

I frankly wasn't aware that centos/rhel didn't support Docker. And sure enough, yeah, we don't run the Docker packaging tests on CentOS. I think we should make an explicit decision here on Podman support. It would mean adapting our packaging tests to support using podman in lieu of docker.

@bytebilly Aside from the existing conversation on CentOS, we do intend to start testing on RHEL (#66305) and it would be identical to our situation today with CentOS, meaning that we would not be testing Docker images on that platform. I'm not sure we even make this distinction in any kind of compatibility matrix (we don't really break it down by packaging type). Is providing this support (and associated testing) a compelling enough thing to do? FYI, if you thin it's best we can add a discuss label and bring this up in the next sync.

[geragio]

Docker now supports also CentOS 8, now it should be possible to run Elastic stack on this operating system

[mark-vieira]

@geragio have you attempted yourself to run the Elasticsearch docker image on CentOS with either docker or podman?

[geragio]

@mark-vieira I am able to run the Elastic stack using the docker-compose file in the official documentation but not with podman-compose. Probably some small adjustments are required in the yaml file

[jhoff909]

I'm running elasticsearch and kibana using podman containers on centos 7 and am unable to get container level os stats using xpack.monitoring.ui.container.elasticsearch.enabled: true although omitting this from the kibana config seems to work fine. Would be nice if Elastic officially supported Podman.

[matsmcp]

Another vote for This. RHEL7 + Redhat Docker 1.13 is old today. RHEL8 + DockerCE isn't a fully supported solution since Redhat doesn't support DockerCE and for some usecases a fully supported stack is a requirement

RHEL8+ Podman would be fully supported and it would solve the old issue with docker running as root

[bytebilly]

Supporting Podman is an interesting option. It's not just about RHEL, but also its entire crew (Fedora, Rocky Linux, Oracle Linux, CentOS). We probably want to figure out implications for our Docker Compose files too.

[geragio]

I was able to run a Elasticsearch cluster using podman and podman-compose with some caveats:

In order to run it I have followed the following steps:

• Download official docker-compose.yml defined in the Elastic documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-compose-file
• Set vm.max_map_count=262144 on my system
• Run podman-compose up -d with root privileges

You can't start an Elasticsearch cluster in rootless mode(which is a quite nice feature of podman) because the official docker-compose.yml overrides ulimits and the task of course is not allowed if you are not root user.

I was able to run the cluster also in rootless mode but I had to apply some changes to the official docker-compose file. Feel free to use the docker-compose file I have used: https://gist.github.com/geragio/32ac8bbcfb0e51e9a2399ae4586730bd

[rahadiancs]

Any new update on this? I was able to run Elasticsearch and Kibana with podman (not using the unofficial podman-compose) on Oracle Linux 8.6. However there's a problem with binding config volume and both container. In Elasticsearch container, I can bind data and logs directory, and in Kibana container I can bind data directory, but not config directory.

I chown 1000:1000 on all of them, but only data and logs works.

My goal is to setup APM server using single host with podman on development environment because I want to keep the server neat and clean.

[dlonghi]

Any new update on this? I was able to run Elasticsearch and Kibana with podman (not using the unofficial podman-compose) on Oracle Linux 8.6. However there's a problem with binding config volume and both container. In Elasticsearch container, I can bind data and logs directory, and in Kibana container I can bind data directory, but not config directory.

I chown 1000:1000 on all of them, but only data and logs works.

My goal is to setup APM server using single host with podman on development environment because I want to keep the server neat and clean.

Did you try to put a :z on the bind-mount paramenter or disable SELinux on the OL 8.6 box ?

[rahadiancs]

Any new update on this? I was able to run Elasticsearch and Kibana with podman (not using the unofficial podman-compose) on Oracle Linux 8.6. However there's a problem with binding config volume and both container. In Elasticsearch container, I can bind data and logs directory, and in Kibana container I can bind data directory, but not config directory. I chown 1000:1000 on all of them, but only data and logs works. My goal is to setup APM server using single host with podman on development environment because I want to keep the server neat and clean.

Did you try to put a :z on the bind-mount paramenter or disable SELinux on the OL 8.6 box ?

I tried both already, and it's still not working. Even changed the permission to 777, also not working. And as I said earlier, I have no problem binding the data and logs directory, only config directory is not bindable.

[Hythloday-zero]

(from the Fed Consulting side of things) I get the feeling that podman wants things to behave more like OpenShift, thus using... well, pods. It also then presumes some level of security context setup, permissions, max_map_count, root, etcetc. Do we have any plans to (or does anyone have) create a working yaml that can be run be podman play?

I found this from redhat's site when trying to fumble through a docker-compose -to- podman-compose fiasco: RedHat Podman conversion

[mark-vieira]

As of right now, explicit Podman support is not on our immediate horizon. The impression I get is that our existing images will work, albeit potentially with some fiddling and a lack of explicit guidance or testing.

[magorbalassy]

From the Redhat Container 101 docs:

There are several competing Container Image formats (Docker, Appc, LXD), but the industry is moving forward with a standard governed under the Open Container Initiative. Our images are OCI compliant.

There are also several competing Container Engines including Docker, CRI-O, Railcar, RKT, LXC. These Container Engines take a Container Image and turn it into a Container (aka running processes).

And finally Tools which target the OCI Container Image Format Specification and Container Runtime Specification ensure portability between a broad ecosystem of container platforms, container engines, and supporting tools across cloud providers and on premise architectures.

If our images are OCI compliant, does that mean that we have the compatibility with the container engines ensured by the OCI standard itself ?

[mark-vieira]

If our images are OCI compliant, does that mean that we have the compatibility with the container engines ensured by the OCI standard itself ?

That's the theory.

[jchorier]

Hi, Do you have any update for that ? I saw that podman is supported in ECE so I thought that perhaps ... https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-using-podman-cloud.html