Running `pytest` very slow

[grzegorzk]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I tried to run pytest from within the container. It takes long time to run small number of tests (7) - to run same tests on host takes milliseconds.

Steps to reproduce the issue:

1. podman run -it --rm ubuntu /bin/bash

2. apt-get update

3. apt-get install python3 python3-pip git

4. pip3 install pytest

5. git clone https://github.com/any_python_repo/with_tests

6. cd with_tests && pip3 install -e .

7. pytest

Describe the results you received:

It took a few seconds to collect and run tests

Describe the results you expected:

Running pytest should take same amount of time on host and on container

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

(paste your output here)

Output of podman info --debug:

debug:
compiler: gc
git commit: 2ced9094d4728dd09f60a177faa32339a8d0f721
go version: go1.13.7
podman version: 1.8.0
host:
BuildahVersion: 1.13.1
CgroupVersion: v1
Conmon:
package: Unknown
path: /usr/bin/conmon
version: 'conmon version 2.0.9, commit: 60b42f20f222df9da07a01ee444327348b9f54ee'
Distribution:
distribution: arch
version: unknown
IDMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
MemFree: 67353055232
MemTotal: 134693437440
OCIRuntime:
name: runc
package: Unknown
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc10
commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
spec: 1.0.1-dev
SwapFree: 0
SwapTotal: 0
arch: amd64
cpus: 12
eventlogger: journald
hostname: archlinux
kernel: 5.5.3-arch1-1
os: linux
rootless: true
slirp4netns:
Executable: /usr/bin/slirp4netns
Package: Unknown
Version: |-
slirp4netns version 0.4.3
commit: 2244b9b6461afeccad1678fac3d6e478c28b4ad6
uptime: 8h 15m 53.55s (Approximately 0.33 days)
registries:
search:
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- registry.centos.org
store:
ConfigFile: /home/grzegorz/.config/containers/storage.conf
ContainerStore:
number: 29
GraphDriverName: overlay
GraphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: Unknown
Version: |-
fusermount3 version: 3.9.0
fuse-overlayfs: version 0.7.5
FUSE library version 3.9.0
using FUSE kernel interface version 7.31
GraphRoot: /home/grzegorz/.local/share/containers/storage
GraphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
ImageStore:
number: 111
RunRoot: /run/user/1000/containers
VolumePath: /home/grzegorz/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

pacman -Q podman
podman 1.8.0-1

Additional environment details (AWS, VirtualBox, physical, etc.):

[baude]

I'm inclined, but won't, to make the claim this is not about podman. Running those tests on the host and inside a container are not fair assessments of podman. You are running rootless, and therefore filesystems are at play that might explain this readily. Lets see what others have to say.

[mheon]

The two big potential causes I can think of:

• Seccomp. Large whitelists are a performance issue all on their own, and added to that they presently unconditionally trigger some additional layers of defense against Spectre-type attacks (which, naturally, slow things down further). This can be disabled (at cost of security) with --security-opt seccomp=unconfined
• Filesystem. Rootless does not have any privileges a normal user does not have, so we can't mount the normal in-kernel overlayfs filesystem; we can, however, mount FUSE filesystems, so we use fuse-overlayfs, a FUSE reimplementation of overlay. This does, however, incur the usual latency issues

[mheon]

I think I would expect the performance of root Podman using kernel overlayfs with Seccomp disabled to be very similar to running outside of a container. Unfortunately, without kernel Overlay support (this is being debated in the kernel community, there are security implications), we can't guarantee that level of performance for rootless Podman. But things will probably be significantly faster if you turn off Seccomp.

[baude]

maybe edit the original post with a real example for step 5 and it could be examined.

[grzegorzk]

Hi @baude, I will build test package over the weekend. Simple dependencies-less package I have already built does not show this problem so I'll try to put the same dependencies.

[rhatdan]

@giuseppe FYI

[grzegorzk]

I tried to build reference package and after spending some time with it I pinpointed the offending package - boto3 - which was expecting ~/.aws directory with some configuration to exist. I was too fast to jump into conclusions, doh! Apologies.