containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.82k stars 2.42k forks source link

non-root podman cannot see images from non-root buildah #1972

Closed ardrigh closed 5 years ago

ardrigh commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

kind bug

Description This is a bit of an odd problem. In Fedora 28 I was able to run non-root buildah and then run non-root podman for a simple image workflow when required.

In Fedora 29 non-root buildah successfully creates images but non-root podman cannot see them.

I notice podman info lists GraphDriverName: vfs when the configs all mention using overlayfs which seems odd.

Steps to reproduce the issue:

export newcontainer=$(buildah from fedora)
echo $newcontainer
buildah run $newcontainer -- dnf install -y git ruby ruby-devel rubygems rubygems-devel gcc redhat-rpm-config make make-devel libffi libffi-devel
buildah run $newcontainer -- gem install travis
buildah run $newcontainer -- /usr/local/bin/travis version
buildah commit $newcontainer traviscli
buildah images
podman images

Describe the results you received: Buildah successfully runs, creates image which is displayed in the list. Buildah stores the new files under ~/.local/share/containers/storage/overlay-images/

Podman cannot see the new image.

Describe the results you expected: Buildah successfully runs and Podman can see the image to run it.

podman run -rm -it --volume /local/path/to/git_repo/:/inside_container/path/to/git_repo:z traviscli /usr/bin/bash

Additional information you deem important (e.g. issue happens only occasionally): I think the system is getting confused between buildah using overlayfs and podman using vfs.

Output of podman version:

Version:       0.12.1.1
Go Version:    go1.11.2
Git Commit:    "0c9259a2c84f6234a29dcd78674c3f4837fd3101"
Built:         Sat Dec  8 08:29:27 2018
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-0.12.1.1-1.git66d3499.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 0c9259a2c84f6234a29dcd78674c3f4837fd3101'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 3178217472
  MemTotal: 16700493824
  OCIRuntime:
    package: runc-1.0.0-59.dev.gitccb5efd.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      commit: 6e5a791a02fefb403034e0de8693d225d52b33a7
      spec: 1.0.1-dev
  SwapFree: 26839347200
  SwapTotal: 26839347200
  arch: amd64
  cpus: 4
  hostname: host.example.com
  kernel: 4.19.7-300.fc29.x86_64
  os: linux
  rootless: true
  uptime: 3h 28m 32.05s (Approximately 0.12 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/user/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/1000/run

Additional environment details (AWS, VirtualBox, physical, etc.):

$ buildah version
Version:         1.5
Go Version:      go1.11.2
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Jan  1 12:00:00 1970
OS/Arch:         linux/amd64
$ cat storage.conf 
RunRoot = "/run/user/1000"
GraphRoot = "/home/user/.local/share/containers/storage"
GraphDriverName = "overlay"
GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]
$ cat libpod.conf 
volume_path = "/home/user/.local/share/containers/storage/volumes"
image_default_transport = "docker://"
runtime_path = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"]
conmon_path = ["/usr/libexec/podman/conmon", "/usr/libexec/crio/conmon", "/usr/local/lib/podman/conmon", "/usr/local/libexec/crio/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/lib/crio/bin/conmon"]
conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
cgroup_manager = "cgroupfs"
static_dir = "/home/user/.local/share/containers/storage/libpod"
tmp_dir = "/run/user/1000/libpod/tmp"
max_log_size = -1
no_pivot_root = false
cni_config_dir = "/etc/cni/net.d/"
cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"]
infra_image = "k8s.gcr.io/pause:3.1"
infra_command = "/pause"
enable_port_reservation = true
label = true
mheon commented 5 years ago

To get the new Overlay driver you may need to delete your local Podman files - we only changed the default from VFS in 0.12.1, and if you used Podman prior to that we keep you on VFS - the swap to overlay is a breaking change.

I suspect this may be part of why you're not seeing the Buildah images. Buildah may be using the new Overlay default while we're stuck on VFS.

On Mon, Dec 10, 2018, 23:45 Dagan McGregor <notifications@github.com wrote:

Is this a BUG REPORT or FEATURE REQUEST?:

kind bug

Description This is a bit of an odd problem. In Fedora 28 I was able to run non-root buildah and then run non-root podman for a simple image workflow when required.

In Fedora 29 non-root buildah successfully creates images but non-root podman cannot see them.

I notice podman info lists GraphDriverName: vfs when the configs all mention using overlayfs which seems odd.

Steps to reproduce the issue:

export newcontainer=$(buildah from fedora) echo $newcontainer buildah run $newcontainer -- dnf install -y git ruby ruby-devel rubygems rubygems-devel gcc redhat-rpm-config make make-devel libffi libffi-devel buildah run $newcontainer -- gem install travis buildah run $newcontainer -- /usr/local/bin/travis version buildah commit $newcontainer traviscli buildah images podman images

Describe the results you received: Buildah successfully runs, creates image which is displayed in the list. Buildah stores the new files under ~/.local/share/containers/storage/overlay-images/

Podman cannot see the new image.

Describe the results you expected: Buildah successfully runs and Podman can see the image to run it.

podman run -rm -it --volume /local/path/to/git_repo/:/inside_container/path/to/git_repo:z traviscli /usr/bin/bash

Additional information you deem important (e.g. issue happens only occasionally): I think the system is getting confused between buildah using overlayfs and podman using vfs.

Output of podman version:

Version: 0.12.1.1 Go Version: go1.11.2 Git Commit: "0c9259a2c84f6234a29dcd78674c3f4837fd3101" Built: Sat Dec 8 08:29:27 2018 OS/Arch: linux/amd64

Output of podman info:

host: BuildahVersion: 1.6-dev Conmon: package: podman-0.12.1.1-1.git66d3499.fc29.x86_64 path: /usr/libexec/podman/conmon version: 'conmon version 1.12.0-dev, commit: 0c9259a2c84f6234a29dcd78674c3f4837fd3101' Distribution: distribution: fedora version: "29" MemFree: 3178217472 MemTotal: 16700493824 OCIRuntime: package: runc-1.0.0-59.dev.gitccb5efd.fc29.x86_64 path: /usr/bin/runc version: |- runc version 1.0.0-rc6 commit: 6e5a791a02fefb403034e0de8693d225d52b33a7 spec: 1.0.1-dev SwapFree: 26839347200 SwapTotal: 26839347200 arch: amd64 cpus: 4 hostname: host.example.com kernel: 4.19.7-300.fc29.x86_64 os: linux rootless: true uptime: 3h 28m 32.05s (Approximately 0.12 days) insecure registries: registries: [] registries: registries:

  • docker.io
  • registry.fedoraproject.org
  • quay.io
  • registry.access.redhat.com
  • registry.centos.org store: ContainerStore: number: 0 GraphDriverName: vfs GraphOptions:
  • overlay.mount_program=/usr/bin/fuse-overlayfs GraphRoot: /home/user/.local/share/containers/storage GraphStatus: {} ImageStore: number: 0 RunRoot: /run/user/1000/run

Additional environment details (AWS, VirtualBox, physical, etc.):

$ buildah version Version: 1.5 Go Version: go1.11.2 Image Spec: 1.0.0 Runtime Spec: 1.0.0 CNI Spec: 0.4.0 libcni Version: Git Commit: Built: Thu Jan 1 12:00:00 1970 OS/Arch: linux/amd64

$ cat storage.conf RunRoot = "/run/user/1000" GraphRoot = "/home/user/.local/share/containers/storage" GraphDriverName = "overlay" GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]

$ cat libpod.conf volume_path = "/home/user/.local/share/containers/storage/volumes" image_default_transport = "docker://" runtime_path = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"] conmon_path = ["/usr/libexec/podman/conmon", "/usr/libexec/crio/conmon", "/usr/local/lib/podman/conmon", "/usr/local/libexec/crio/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/lib/crio/bin/conmon"] conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"] cgroup_manager = "cgroupfs" static_dir = "/home/user/.local/share/containers/storage/libpod" tmp_dir = "/run/user/1000/libpod/tmp" max_log_size = -1 no_pivot_root = false cni_config_dir = "/etc/cni/net.d/" cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"] infra_image = "k8s.gcr.io/pause:3.1" infra_command = "/pause" enable_port_reservation = true label = true

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/containers/libpod/issues/1972, or mute the thread https://github.com/notifications/unsubscribe-auth/AHYHCAhS4T8Jj6-IyVqRpT4MpKo68rCqks5u3zh1gaJpZM4ZMqPP .

ardrigh commented 5 years ago

Fedora 29 comes with v0.10 of Podman pre-installed, and the version I am running I pulled from updates-testing. I suspect a number of people will run into similar issues to me.

It's not clear why deleting all of the existing files would fix this problem, when the files being used were pointed at using overlayfs.

The files in ~/.config/containers that I copied into the ticket don't point anywhere for Podman to keep using VFS even after a package upgrade.

The line in ~/.config/containers/storage.conf clearly states: GraphDriverName = "overlay"

And Podman was ignoring it.

Having removed both the buildah and podman packages are deleting all associated directories and file, the reinstall has picked up to use overlay as the driver as expected. It still seems like a bug.

I do note that the format of ~/.config/containers/storage.conf has now changed:

$ cat storage.conf 
[storage]
  driver = "overlay"
  runroot = "/run/user/1000"
  graphroot = "/home/user/.local/share/containers/storage"
  [storage.options]
    mount_program = "/usr/bin/fuse-overlayfs"
ardrigh commented 5 years ago

As a comparison, after removing the packages and installing again, this is the new output for podman info

podman info
host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-0.12.1.1-1.git66d3499.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 0c9259a2c84f6234a29dcd78674c3f4837fd3101'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 12967436288
  MemTotal: 16700493824
  OCIRuntime:
    package: runc-1.0.0-59.dev.gitccb5efd.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      commit: 6e5a791a02fefb403034e0de8693d225d52b33a7
      spec: 1.0.1-dev
  SwapFree: 26839347200
  SwapTotal: 26839347200
  arch: amd64
  cpus: 4
  hostname: host.example.com
  kernel: 4.19.7-300.fc29.x86_64
  os: linux
  rootless: true
  uptime: 16m 57.89s
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/user/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 2
  RunRoot: /run/user/1000

I am not sure why this is listed twice now though:

  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  - overlay.mount_program=/usr/bin/fuse-overlayfs

That line is commented out in the file in /etc/containers/storage.conf and it's not in the other config files I can see.

giuseppe commented 5 years ago

@ardrigh there was an issue in the previous version of Podman and it was generating the wrong storage.conf file. The new format is the correct one.

Are both Buildah and Podman using fuse-overlayfs now?

ardrigh commented 5 years ago

Yes, after clearing the files and doing the reinstall, podman images now successfully lists the images created by buildah.

Thanks.

mheon commented 5 years ago

I think setting vfs when storage.conf explicitly states overlay is something we do need to look into; I'm hesitant to respect storage.conf unconditionally given our prior issues with Rootless breaking changes, but printing a warning (Overriding storage driver overlay as database was created with VFS driver or similar) would be a good thing.

giuseppe commented 5 years ago

can this be closed now?

mheon commented 5 years ago

Since the DB now detects path changes I think this is safe to close