dsavineau
commented
5 years ago For example edit unit for mgr, comment After=docker.service and replace /usr/bin/docker with /usr/bin/podman. Attempt to restart.
This is true for stable-3.2 but we don't support podman on that branch. The stable-4.0 already supports this (so it's nautilus only).
Podman container started with ceph daemon runing. BTW, when I run podman command without systemd unit it works, i.e. starts runs, no stderr problem.
Honestly we currently test podman on RHEL 8 so it's possible that the 1.3.0 release also change something
alitvak69
Couldn't find anything explaining the possible problem
Bug Report
What happened: Starting systemd for ceph daemons using the same unit file but changing docker to podman
For some reason, I am failing to start podman mon container on a second node or third node
May 9 06:13:05 storage2n2-la.xcastlabs.net systemd[1]: Starting Ceph Monitor... May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38307]: Error: no container with name or ID ceph-mon-storage2n2-la found: no such container May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: Started Ceph Monitor. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Using graph driver overlay" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Using graph root /var/lib/containers/storage" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Using run root /var/run/containers/storage" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Using static dir /var/lib/containers/storage/libpod" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Using tmp dir /var/run/libpod" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Using volume path /var/lib/containers/storage/volumes" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Set libpod namespace to """ May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="[graphdriver] trying provided driver "overlay"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="overlay test mount with multiple lowers succeeded" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="overlay test mount indicated that metacopy is not being used" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=info msg="Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.xcastlabs.net/ceph/daemon:latest-nautilus-centos-7-x86_64"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@ac2018decd3f7d8bb4ab58bdbc6bd55b942bc4e5719699ec5057c33fb73b6bbd"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="exporting opaque data as blob "sha256:ac2018decd3f7d8bb4ab58bdbc6bd55b942bc4e5719699ec5057c33fb73b6bbd"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@ac2018decd3f7d8bb4ab58bdbc6bd55b942bc4e5719699ec5057c33fb73b6bbd"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="exporting opaque data as blob "sha256:ac2018decd3f7d8bb4ab58bdbc6bd55b942bc4e5719699ec5057c33fb73b6bbd"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@ac2018decd3f7d8bb4ab58bdbc6bd55b942bc4e5719699ec5057c33fb73b6bbd"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Using host netmode" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="User mount /var/lib/ceph:/var/lib/ceph options [z]" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="User mount /etc/ceph:/etc/ceph options [z]" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="User mount /var/run/ceph:/var/run/ceph options [z]" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="User mount /etc/localtime:/etc/localtime options [ro]" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Adding mount /proc" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Adding mount /dev" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Adding mount /dev/pts" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Adding mount /dev/mqueue" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Adding mount /sys" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Adding mount /sys/fs/cgroup" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="appending name ceph-mon-storage2n2-la" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Allocated lock 0 for container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@ac2018decd3f7d8bb4ab58bdbc6bd55b942bc4e5719699ec5057c33fb73b6bbd"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="exporting opaque data as blob "sha256:ac2018decd3f7d8bb4ab58bdbc6bd55b942bc4e5719699ec5057c33fb73b6bbd"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="created container "e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="container "e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" has work directory "/var/lib/containers/storage/overlay-containers/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/userdata"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="container "e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" has run directory "/var/run/containers/storage/overlay-containers/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/userdata"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="New container created "e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="container "e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" has CgroupParent "machine.slice/libpod-e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9.scope"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Not attaching to stdin" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/O66PJB5V7CRSKAXSQZS7GVVVHP:/var/lib/containers/storage/overlay/l/NK3VBVEQDJLEQ5A5HDGPYUWQ56:/var/lib/containers/storage/overlay/l/LHFWJRWX2W4LWSJQ6VJ5XJGLH2:/var/lib/containers/storage/overlay/l/OU3PDDNORAJMGHL6FBKJNNZDTZ:/var/lib/containers/storage/overlay/l/3Z23EUG2HU3CBME3IQLG2DERSV:/var/lib/containers/storage/overlay/l/RGVUZOMXTGC6CISP4M6ODXJHYO:/var/lib/containers/storage/overlay/l/SOOTXOJIWBKEHHJITGB7OG45DG:/var/lib/containers/storage/overlay/l/BJHBVUIHUJ4FXRD6TTKH2D55GE:/var/lib/containers/storage/overlay/l/MOTUNMC3KWYFU2P5LOVX2ZTREJ:/var/lib/containers/storage/overlay/l/SABMFPC2OKP53MP64EXJAC72SX,upperdir=/var/lib/containers/storage/overlay/60ccedc7641ef31f71e687022b9bbe7313a7c9c8f4d6b141bf8ff9d3dc669081/diff,workdir=/var/lib/containers/storage/overlay/60ccedc7641ef31f71e687022b9bbe7313a7c9c8f4d6b141bf8ff9d3dc669081/work" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="mounted container "e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" at "/var/lib/containers/storage/overlay/60ccedc7641ef31f71e687022b9bbe7313a7c9c8f4d6b141bf8ff9d3dc669081/merged"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Created root filesystem for container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 at /var/lib/containers/storage/overlay/60ccedc7641ef31f71e687022b9bbe7313a7c9c8f4d6b141bf8ff9d3dc669081/merged" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="/etc/system-fips does not exist on host, not mounting FIPS mode secret" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Setting CGroups for container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 to machine.slice:libpod:e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="reading hooks from /usr/share/containers/oci/hooks.d" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="reading hooks from /etc/containers/oci/hooks.d" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Created OCI spec for container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 at /var/lib/containers/storage/overlay-containers/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/userdata/config.json" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="/usr/libexec/crio/conmon messages will be logged to syslog" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="running conmon: /usr/libexec/crio/conmon" args=[-s -c e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 -u e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/userdata -p /var/run/containers/storage/overlay-containers/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/userdata/pidfile -l /var/lib/containers/storage/overlay-containers/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/userdata/ctr.log --exit-dir /var/run/libpod/exits --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 --socket-dir-path /var/run/libpod/socket --log-level debug --syslog] May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=info msg="Running conmon under slice machine.slice and unitName libpod-conmon-e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9.scope" May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: Started libpod-conmon-e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9.scope. May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: Created slice libcontainer_38450_systemd_test_default.slice. May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: Removed slice libcontainer_38450_systemd_test_default.slice. May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: Created slice libcontainer_38450_systemd_test_default.slice. May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: Removed slice libcontainer_38450_systemd_test_default.slice. May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: Started libcontainer container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9. May 9 06:13:06 storage2n2-la.xcastlabs.net conmon: conmon e1db80f2911027bb98d9 : container PID: 38482 May 9 06:13:06 storage2n2-la.xcastlabs.net conmon: conmon e1db80f2911027bb98d9 : attach sock path: /var/run/libpod/socket/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/attach May 9 06:13:06 storage2n2-la.xcastlabs.net conmon: conmon e1db80f2911027bb98d9 : addr{sun_family=AF_UNIX, sun_path=/var/run/libpod/socket/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/attach} May 9 06:13:06 storage2n2-la.xcastlabs.net conmon: conmon e1db80f2911027bb98d9 : ctl fifo path: /var/lib/containers/storage/overlay-containers/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/userdata/ctl May 9 06:13:06 storage2n2-la.xcastlabs.net conmon: conmon e1db80f2911027bb98d9 : terminal_ctrl_fd: 16 May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Received container pid: 38482" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Created container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 in OCI runtime" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Attaching to container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="connecting to socket /var/run/libpod/socket/e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9/attach" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Starting container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 with command [/opt/ceph-container/bin/entrypoint.sh]" May 9 06:13:06 storage2n2-la.xcastlabs.net conmon: conmon e1db80f2911027bb98d9 : Accepted connection 18 May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Enabling signal proxying" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: open /dev/stderr: no such device or address May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: NAME: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc - Open Container Initiative runtime May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc is a command line client for running applications packaged according to May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: the Open Container Initiative (OCI) format and is a compliant implementation of the May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: Open Container Initiative specification. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc integrates well with existing process supervisors to provide a production May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: container runtime environment for applications. It can be used with your May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: existing process monitoring tools and the container will be spawned as a May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: direct child of the process supervisor. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: Containers are configured using bundles. A bundle for a container is a directory May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: that includes a specification file named "config.json" and a root filesystem. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: The root filesystem contains the contents of the container. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: To start a new instance of a container: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: # runc run [ -b bundle ] May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: Where "" is your name for the instance of the container that you May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: are starting. The name you provide for the container instance must be unique on May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: your host. Providing the bundle directory using "-b" is optional. The default May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: value for "bundle" is the current directory. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: USAGE: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc [global options] command [command options] [arguments...] May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: VERSION: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: spec: 1.0.1-dev May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: COMMANDS: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: checkpoint checkpoint a running container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: create create a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: delete delete any resources held by the container often used with detached container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: events display container events such as OOM notifications, cpu, memory, and IO usage statistics May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: exec execute new process inside the container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: init initialize the namespaces and launch the process (do not call it outside of runc) May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: kill kill sends the specified signal (default: SIGTERM) to the container's init process May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: list lists containers started by runc with the given root May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: pause pause suspends all processes inside the container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: ps ps displays the processes running inside a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: restore restore a container from a previous checkpoint May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: resume resumes all processes that have been previously paused May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: run create and run a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: spec create a new specification file May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: start executes the user defined process in a created container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: state output the state of a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: update update container resource constraints May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: help, h Shows a list of commands or help for one command May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: GLOBAL OPTIONS: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --debug enable debug output for logging May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --log value set the log file path where internal debug information is written (default: "/dev/stderr") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --log-format value set the format used by logs ('text' (default), or 'json') (default: "text") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --root value root directory for storage of container state (this should be located in tmpfs) (default: "/run/runc") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --criu value path to the criu binary used for checkpoint and restore (default: "criu") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --systemd-cgroup enable systemd cgroup support, expects cgroupsPath to be of form "slice:prefix:name" for e.g. "system.slice:runc:434234" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --rootless value ignore cgroup permission errors ('true', 'false', or 'auto') (default: "auto") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --help, -h show help May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --version, -v print the version May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=error msg="open /dev/stderr: no such device or address\n" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: open /dev/stderr: no such device or address May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Cleaning up container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Network is already cleaned up, skipping..." May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="unmounted container "e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9"" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=debug msg="Container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 storage is already unmounted, skipping..." May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: open /dev/stderr: no such device or address May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: NAME: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc - Open Container Initiative runtime May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc is a command line client for running applications packaged according to May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: the Open Container Initiative (OCI) format and is a compliant implementation of the May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: Open Container Initiative specification. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc integrates well with existing process supervisors to provide a production May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: container runtime environment for applications. It can be used with your May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: existing process monitoring tools and the container will be spawned as a May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: direct child of the process supervisor. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: Containers are configured using bundles. A bundle for a container is a directory May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: that includes a specification file named "config.json" and a root filesystem. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: The root filesystem contains the contents of the container. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: To start a new instance of a container: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: # runc run [ -b bundle ] May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: Where "" is your name for the instance of the container that you May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: are starting. The name you provide for the container instance must be unique on May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: your host. Providing the bundle directory using "-b" is optional. The default May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: value for "bundle" is the current directory. May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: USAGE: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: runc [global options] command [command options] [arguments...] May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: VERSION: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: spec: 1.0.1-dev May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: COMMANDS: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: checkpoint checkpoint a running container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: create create a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: delete delete any resources held by the container often used with detached container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: events display container events such as OOM notifications, cpu, memory, and IO usage statistics May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: exec execute new process inside the container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: init initialize the namespaces and launch the process (do not call it outside of runc) May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: kill kill sends the specified signal (default: SIGTERM) to the container's init process May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: list lists containers started by runc with the given root May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: pause pause suspends all processes inside the container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: ps ps displays the processes running inside a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: restore restore a container from a previous checkpoint May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: resume resumes all processes that have been previously paused May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: run create and run a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: spec create a new specification file May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: start executes the user defined process in a created container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: state output the state of a container May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: update update container resource constraints May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: help, h Shows a list of commands or help for one command May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: GLOBAL OPTIONS: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --debug enable debug output for logging May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --log value set the log file path where internal debug information is written (default: "/dev/stderr") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --log-format value set the format used by logs ('text' (default), or 'json') (default: "text") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --root value root directory for storage of container state (this should be located in tmpfs) (default: "/run/runc") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --criu value path to the criu binary used for checkpoint and restore (default: "criu") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --systemd-cgroup enable systemd cgroup support, expects cgroupsPath to be of form "slice:prefix:name" for e.g. "system.slice:runc:434234" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --rootless value ignore cgroup permission errors ('true', 'false', or 'auto') (default: "auto") May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --help, -h show help May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: --version, -v print the version May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=error msg="open /dev/stderr: no such device or address\n" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: open /dev/stderr: no such device or address May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=error msg="unable to remove container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 after failing to start and attach to it" May 9 06:13:06 storage2n2-la.xcastlabs.net podman[38374]: time="2019-05-09T06:13:06-07:00" level=error msg="error attaching to container e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9: /usr/bin/runc start e1db80f2911027bb98d90f620d5f14dba3963de03c98b42512c5b00c70e96ae9 failed: exit status 1" May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: ceph-mon@storage2n2-la.service: Main process exited, code=exited, status=127/n/a May 9 06:13:06 storage2n2-la.xcastlabs.net systemd[1]: ceph-mon@storage2n2-la.service: Failed with result 'exit-code'. What you expected to happen: Podman container started with ceph daemon runing. BTW, when I run podman command without systemd unit it works, i.e. starts runs, no stderr problem.
How to reproduce it (minimal and precise):
For example edit unit for mgr, comment After=docker.service and replace /usr/bin/docker with /usr/bin/podman. Attempt to restart.
Share your group_vars files, inventory
Environment:
uname -a): 4.18.0.14docker version): podman 1.3.0ansible-playbook --version): 2.7git head or tag or stable branch): stable-3.2 and stable-4.0 (tried both)ceph -v): Mimic and Nautilus (tried both)