Java OOM with docker, not with podman

[stellirin]

Testing a container with a java based app I found that certain editions of java do not start on Fedora CoreOS when run with docker. Of the four commands below, the first three show the help text, whereas the last command shows library initialization failed - unable to allocate file descriptor table - out of memory.

sudo podman run -it --rm adoptopenjdk:8-jre-hotspot java
sudo podman run -it --rm adoptopenjdk:8-jre-openj9 java
sudo docker run -it --rm adoptopenjdk:8-jre-hotspot java
sudo docker run -it --rm adoptopenjdk:8-jre-openj9 java

It surprised me that the container image works with podman but not with docker on FCOS. 😂

Both container images works with Docker Desktop on Windows.

(Actually I was testing the container launch from a systemd service, so I wasted quite some hours trying to debug in the wrong place, thinking it was my badly written ignition file.)

rpm-ostree status

State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 30.20190905.0 (2019-09-05T14:36:50Z)
                    Commit: b4beca154dab3696fd04f32ddab818102caa9247ec3192403adb9aaecc991bd9
              GPGSignature: (unsigned)

[stellirin]

So it seems like a classic ulimit 'issue', I was able to make it work with --ulimit nofile=65536:65536 in the docker run command.

I'm not sure if this is something that should be addressed? I know this is something that is typically more restricted on RHEL/CentOS, perhaps this is working 'as intended'.

[lucab]

@stellirin thanks for the report and glad that you figured it out.

We acknowledge the fact that different container runtimes have different semantics and defaults, and we normally just try to point users at relevant documentation so that they are aware of the specific behavior.

For docker, the default limits are documented at https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container---ulimit. For podman, so far I've only found https://github.com/containers/libpod/blob/v1.5.1/docs/podman-run.1.md which however does not mention defaults nor CLI flag syntax.

[dustymabe]

cc @mheon @baude in case they have anything to add to the discussion.

[mheon]

We hardcode a default ulimit of 65536 for files unconditionally in Podman, added for Docker compatibility. It looks like Docker's ulimit must somehow be conditional, where ours is universal (for root containers. We don't have enough permissions to set high ulimits with rootless).

[stellirin]

For now I can live with the work around, I need to use Docker at least until Podman learns the --network option so I can have isolated containers talk to each other using names.

[dustymabe]

Thanks all.. I take it that we don't have any further action to take from this ticket? I'm going to close now. Please re-open if there is further action to take.

[masterzen]

We had this issue with one of our server container, because the default FCOS /etc/sysconfig/docker contains the default ulimit docker options to limit nofile to 1024...

[bgilbert]

@masterzen Good catch. Reopening.

[dustymabe]

@bgilbert @olivierlemasle is there anything for us to do here? Is the proposal that we try to change /etc/sysconfig/docker from the built moby-engine RPM to be higher or to not set it at all.

[bgilbert]

Yes, one of those. It seems strange that the package is intentionally setting a low value.

[travier]

Was introduced in https://src.fedoraproject.org/rpms/moby-engine/c/a6ffb4992d9f134013d2b7a0320e7ad1643230b2?branch=rawhide. Might need a bug report to figure out why this was added.

[bgilbert]

The commit message references https://bugzilla.redhat.com/show_bug.cgi?id=1715254 and https://bugzilla.redhat.com/show_bug.cgi?id=1708115.

[olivierlemasle]

Hi @bgilbert,

Yes, I added this ulimit as suggested in https://bugzilla.redhat.com/show_bug.cgi?id=1715254, just after I took over the moby-engine package (which was orphaned then).

I'm open to increase the ulimit value.