Running detached containers concurrently takes 2x/3x time longer than docker runtime does

[bogdando]

kind bug

Description

Concurrent starting of containers with podman takes longer, sometimes up to 5x times, than docker.

Steps to reproduce the issue:

Fedora 28, 4.18.7-200.fc28.x86_64 docker version 1.13.1, build 1556cce-unsupported podman version 0.10.1

1. Run with defaults (as root user) https://gist.github.com/bogdando/9b6dc40c06aa3b8cc70b0cee120aae84

Example results https://pastebin.com/vEJGkHFu

2. Note the time it takes for each container to get started (ignore the summary time shown at the end of each of the full podman / docker run as it shows something different than what is expected)

3. Compare individual results for podman vs docker containers

Describe the results you received:

Podman looks 2x-5x times slower, probably because of the strict locking used on its state DB? And it also eats some of the containers, like:

# podman ps -aq --filter name=c | wc -l
99
# docker ps -aq --filter name=c | wc -l
100

Describe the results you expected:

Would be nice to have it not more than 1.5x times slower or equal

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:       0.10.1
Go Version:    go1.10.4
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: podman-0.10.1-1.gite4a1553.fc28.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 452812fc5539855eb472129ff1132919a0e756be-dirty'
  Distribution:
    distribution: fedora
    version: "28"
  MemFree: 11400196096
  MemTotal: 12590895104
  OCIRuntime:
    package: runc-1.0.0-51.dev.gitfdd8055.fc28.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 3
  hostname: localhost.localdomain
  kernel: 4.18.7-200.fc28.x86_64
  os: linux
  uptime: 2h 49m 57.94s (Approximately 0.08 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 105
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

docker info:

Containers: 100
 Running: 0
 Paused: 0
 Stopped: 100
Images: 4
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: 1556cce7e5c5349fdffeb123c092a7681769866a (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 1556cce7e5c5349fdffeb123c092a7681769866a-dirty (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 4.18.7-200.fc28.x86_64
Operating System: Fedora 28 (Cloud Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 3
Total Memory: 11.73 GiB
Name: localhost.localdomain
ID: 64YB:COXG:5OPR:PLLR:MBLV:T3CB:D3T7:YIQH:P7QG:7ZG7:HNJZ:JLH5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true
Registries: docker.io (secure), registry.fedoraproject.org (secure), quay.io (secure), registry.access.redhat.com (secure), registry.centos.org (secure), docker.io (secure)

[mheon]

Part of me wonders if Docker isn't hiding some latency in the daemon - you don't need to keep the CLI frontend process around as long as we do, it can exit once you're confident the start request has been sent without error.

Also, if you need performance here, you can run Podman in parallel easily, which should significantly speed things up.

More concerning to me is that it's seemingly eating containers for you - are you seeing any errors?

[mheon]

To clarify further: We have done some tests on parallelizing some container operations (stop, for example, happens in parallel in 0.10.1 by default, running a number of worker threads to stop in parallel). We're seeing good speedups, even on code that seems to make heavy use of locking as run does, but we don't have any solid results for run yet.

We're also not opposed to improving our speed in general, but right now we've been getting a lot of mileage our of running operations in parallel instead of serially

[baude]

Do people really feel detached vs non-detached is a part of the equation?

[mheon]

It's mostly relevant in that it force Docker/Podman to remain alive for the duration the container is alive. Could be relevant if Docker is returning earlier than we are, could be nothing but the extra bit of latency from attaching to the container

[bogdando]

@mheon

Part of me wonders if Docker isn't hiding some latency in the daemon

it totally might be, that's why I did not report the equal times as expected but 1.5 times slower or so :) But even visually one can note the reaction time is longer, especially if the container operations are interleaved with podman ps command, the latter may hang quite for a while before returning something. So if there is some deployment tools, which idempotency is relying on the ps results, it becomes an issue. For example, while the script is running, I can see podman ps hangs for 10s sometimes.

you can run Podman in parallel easily

could you please elaborate on that?

it's seemingly eating containers for you - are you seeing any errors?

The stderr redirection for the benchmark script needs some fixes as it seems got broken in the last revisions of my gist indeed. And I didn't find errors logged to console (for earlier script revisions/podman versions) in journalctl... Perhaps it would be nice to just try it for yourself.

[bogdando]

@baude

Do people really feel detached vs non-detached is a part of the equation?

My benchmark is far from ideal, just noticed that for non-detached containers the time command reports a function of the sleep argument used for the container entry command :)

[bogdando]

specifically, for OpenStack, this issue is a deal breaker, see https://bugs.launchpad.net/tripleo/+bug/1797525/comments/12

[mheon]

To provide a bit of an update here, @baude has begun identifying the worst offenders in create and start performance, and we'll start looking at those in the coming weeks

[baude]

we have made significant progress with run stuff, mind trying master?

[bogdando]

indeed, the same tests repeated with podman version 0.10.2-dev shows very good results for performance at least. thanks!

[EmilienM]

I think this one can be closed now.