containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.98k stars 2.43k forks source link

podman exec failing with Error: AppArmor not initialized correctly: OCI runtime error #6541

Closed develroo closed 4 years ago

develroo commented 4 years ago

/kind bug

Description

After building a podman image and starting it. It starts fine but when trying to exec to it I get an error.

podman exec -it condescending_shamir ls
Error: AppArmor not initialized correctly: OCI runtime error

The debug output is:

podman exec --log-level=debug  -it condescending_shamir ls
DEBU[0000] Reading configuration file "/etc/containers/libpod.conf" 
DEBU[0000] Merged system config "/etc/containers/libpod.conf": &{{false false false false false true} 0 {   [] [] []}  docker://  crun map[crun:[/usr/bin/crun] runc:[/usr/sbin/runc]] [crun runc] [crun] [] [/usr/bin/conmon /usr/sbin/conmon /usr/libexec/podman/conmon /usr/local/libexec/crio/conmon /usr/lib/podman/bin/conmon /usr/libexec/crio/conmon /usr/lib/crio/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd   /var/run/libpod -1 false /etc/cni/net.d/ [/usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman []   k8s.gcr.io/pause:3.1 /pause false false  2048 shm    false} 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver                           
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/sbin/runc"               
INFO[0000] Found CNI network podman (type=ptp) at /etc/cni/net.d/87-podman-ptp.conflist 
DEBU[0000] Handling terminal attach                     
DEBU[0000] Creating new exec session in container ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642 with session id d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -s -c ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642 -u d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642/userdata/d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd -p /var/lib/containers/storage/overlay-containers/ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642/userdata/d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd/exec_pid -l k8s-file:/var/lib/containers/storage/overlay-containers/ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642/userdata/d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd/exec_log --exit-dir /var/lib/containers/storage/overlay-containers/ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642/userdata/d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd/exit --socket-dir-path /var/run/libpod/socket --log-level debug --syslog -t -e --exec-attach --exec-process-spec /var/lib/containers/storage/overlay-containers/ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642/userdata/d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd/exec-process-901009122]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642.scope 
WARN[0000] Failed to add conmon to systemd sandbox cgroup: Unit libpod-conmon-ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642.scope already exists. 
DEBU[0000] Attaching to container ab468d26240446ea00315bec70d945a050f1e38caab096b6fa8dfc158023f642 exec session d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd 
DEBU[0000] connecting to socket /var/run/libpod/socket/d2a3feacdc839fe875eb3d1f171913d1601e145d0100059faea7e93e884f83fd/attach 
DEBU[0000] Received: 0                                  
DEBU[0000] Received a resize event: {Width:213 Height:32} 
DEBU[0000] Received: -256                               
ERRO[0000] [conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent: OCI runtime error 
WARN[0000] unable to find /etc/containers/registries.conf. some podman (image shortnames) commands may be limited 

Output of podman version:

Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.14.3
OS/Arch:            linux/386

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.14.3
  podman version: 1.6.4
host:
  BuildahVersion: 1.11.6
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.9, commit: unknown'
  Distribution:
    distribution: debian
    version: "10"
  MemFree: 28921856
  MemTotal: 452386816
  OCIRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 4231524352
  SwapTotal: 4294963200
  arch: "386"
  cpus: 1
  eventlogger: journald
  hostname: openfire
  kernel: 4.19.0-8-686-pae
  os: linux
  rootless: false
  uptime: 1828h 32m 51.65s (Approximately 76.17 days)
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 30
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unstable,now 1.6.4+dfsg1-3 i386 [installed]

apprmor is installed but no profile for podman seems present.


ii  apparmor                             2.13.2-10                    i386         user-space parser utility for AppArmor
ii  apparmor-profiles-extra              1.26                         all          Extra profiles for AppArmor Security policies
ii  apparmor-utils                       2.13.2-10                    i386         utilities for controlling AppArmor
ii  libapparmor1:i386                    2.13.2-10                    i386         changehat AppArmor library
ii  python3-apparmor                     2.13.2-10                    i386         AppArmor Python3 utility library
ii  python3-libapparmor                  2.13.2-10                    i386         AppArmor library Python3 bindings

Any insights would be appreciated.

mheon commented 4 years ago

@vrothberg PTAL

vrothberg commented 4 years ago

@onlyjob mentioned AppArmor issues on Debian recently. @onlyjob, do you know if they are related?

rhatdan commented 4 years ago

Isn't this a case where the containers-default file that container.conf defaults to is not installed.

develroo commented 4 years ago

Is that supposed to be installed via the package script? I only have two files in /etc/containers.

policy.json

 {
     "default": [
         {
             "type": "insecureAcceptAnything"
         }
     ]
 }

and libpod.conf

 # libpod.conf(5) is the default configuration file for all tools using
 # libpod to manage containers

 # Default transport method for pulling and pushing for images
 image_default_transport = "docker://"

 # Paths to look for the conmon container manager binary.
 # If the paths are empty or no valid path was found, then the `$PATH`
 # environment variable will be used as the fallback.
 conmon_path = [
     "/usr/bin/conmon",
     "/usr/sbin/conmon",
     "/usr/libexec/podman/conmon",
     "/usr/local/libexec/crio/conmon",
     "/usr/lib/podman/bin/conmon",
     "/usr/libexec/crio/conmon",
     "/usr/lib/crio/bin/conmon"
 ]

 # Environment variables to pass into conmon
 conmon_env_vars = [
     "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
 ]

 # CGroup Manager - valid values are "systemd" and "cgroupfs"
 cgroup_manager = "systemd"

 # Container init binary
 #init_path = "/usr/bin/tini"
 #init_path = "/usr/bin/tini-static"
 #init_path = "/usr/bin/dumb-init"
 #init_path = "/usr/bin/catatonit"

 # Directory for persistent libpod files (database, etc)
 # By default, this will be configured relative to where containers/storage
 # stores containers
 # Uncomment to change location from this default
 #static_dir = "/var/lib/containers/storage/libpod"

 # Directory for temporary files. Must be tmpfs (wiped after reboot)
 tmp_dir = "/var/run/libpod"

 # Maximum size of log files (in bytes)
 # -1 is unlimited
 max_log_size = -1

 # Whether to use chroot instead of pivot_root in the runtime
 no_pivot_root = false

 # Directory containing CNI plugin configuration files
 cni_config_dir = "/etc/cni/net.d/"

 # Directories where the CNI plugin binaries may be located
 cni_plugin_dir = [
     "/usr/lib/cni",
     "/usr/local/lib/cni",
     "/opt/cni/bin"
 ]

 # Default CNI network for libpod.
 # If multiple CNI network configs are present, libpod will use the network with
 # the name given here for containers unless explicitly overridden.
 # The default here is set to the name we set in the
 # 87-podman-bridge.conflist included in the repository.
 # Not setting this, or setting it to the empty string, will use normal CNI
 # precedence rules for selecting between multiple networks.
 cni_default_network = "podman"

 # Default libpod namespace
 # If libpod is joined to a namespace, it will see only containers and pods
 # that were created in the same namespace, and will create new containers and
 # pods in that namespace.
 # The default namespace is "", which corresponds to no namespace. When no
 # namespace is set, all containers and pods are visible.
 #namespace = ""

 # Default infra (pause) image name for pod infra containers
 infra_image = "k8s.gcr.io/pause:3.1"

 # Default command to run the infra container
 infra_command = "/pause"

 # Determines whether libpod will reserve ports on the host when they are
 # forwarded to containers. When enabled, when ports are forwarded to containers,
 # they are held open by conmon as long as the container is running, ensuring that
 # they cannot be reused by other programs on the host. However, this can cause
 # significant memory usage if a container has many ports forwarded to it.
 # Disabling this can save memory.
 #enable_port_reservation = true

 # Default libpod support for container labeling
 # label=true

 # The locking mechanism to use
 lock_type = "shm"

 # Number of locks available for containers and pods.
 # If this is changed, a lock renumber must be performed (e.g. with the
 # 'podman system renumber' command).
 num_locks = 2048

 # Directory for libpod named volumes.
 # By default, this will be configured relative to where containers/storage
 # stores containers.
 # Uncomment to change location from this default.
 #volume_path = "/var/lib/containers/storage/volumes"

 # Selects which logging mechanism to use for Podman events.  Valid values
 # are `journald` or `file`.
 # events_logger = "journald"

 # Specify the keys sequence used to detach a container.
 # Format is a single character [a-Z] or a comma separated sequence of
 # `ctrl-<value>`, where `<value>` is one of:
 # `a-z`, `@`, `^`, `[`, `\`, `]`, `^` or `_`
 #
 # detach_keys = "ctrl-p,ctrl-q"

 # Default OCI runtime
 runtime = "crun"
 #runtime = "runc"

 # List of the OCI runtimes that support --format=json.  When json is supported
 # libpod will use it for reporting nicer errors.
 runtime_supports_json = ["crun", "runc"]

 # List of all the OCI runtimes that support --cgroup-manager=disable to disable
 # creation of CGroups for containers.
 runtime_supports_nocgroups = ["crun"]

 # Paths to look for a valid OCI runtime (runc, runv, etc)
 # If the paths are empty or no valid path was found, then the `$PATH`
 # environment variable will be used as the fallback.
 [runtimes]

 runc = [
     "/usr/sbin/runc",
 ]

 crun = [
     "/usr/bin/crun"
 ]

 # The [runtimes] table MUST be the last thing in this file.
 # (Unless another table is added)
 # TOML does not provide a way to end a table other than a further table being
 # defined, so every key hereafter will be part of [runtimes] and not the main
 # config.

does that help?

vrothberg commented 4 years ago

Isn't this a case where the containers-default file that container.conf defaults to is not installed.

I don't think so. The odd thing is that we can podman run but not podman exec.

vrothberg commented 4 years ago

@giuseppe, could this be crun?

giuseppe commented 4 years ago

yes, it is already fixed upstream: https://github.com/containers/crun/pull/391

I'll cut a release hopefully this week.