search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.16k stars 2.36k forks source link

mount image volume inside a podman container #10397

Closed Saini01Basu closed 3 years ago

Saini01Basu commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. podman run --rm -t --mount type=image,source=/var/lib/jenkins/.local/share/containers/storage/overlay-images/0bfbbb5fe1dba30662287087ac0af3093020d521e50a1e6afedc8d54c306b4b0,destination=/opt/pas-goat-appsec,rw=true --image-volume=ignore harbor..com/****/scan-automation:ee6898aa /bin/bash -c 'twistcli images scan --address https://containersecurity.****.com/ --user --password --project PKS --details --podman-path podman pas-goat-appsec'

  2. The container i am running using image - harbor..com/****/scan-automation:ee6898aa has podman installed in it

  3. twistcli images scan just need the image within the above container to start the scan hence trying mount image volume

Describe the results you received:

time="2021-05-19T02:27:23-04:00" level=error msg="error cleaning up container 9547f93a15b4df3104b4192d15bbc6b2097c4acbc32d092a434c29878832c7b4: invalid reference format"

Error: error creating image volume "/var/lib/jenkins/.local/share/containers/storage/overlay-images/0bfbbb5fe1dba30662287087ac0af3093020d521e50a1e6afedc8d54c306b4b0":"/opt/pas-goat-appsec": invalid reference format

Additional information you deem important (e.g. issue happens only occasionally):

**Output of podman info --debug:

11:51:35  + podman info
11:51:35  host:
11:51:35    arch: amd64
11:51:35    buildahVersion: 1.18.0
11:51:35    cgroupManager: cgroupfs
11:51:35    cgroupVersion: v1
11:51:35    conmon:
11:51:35      package: conmon-2.0.22-3.module+el8.3.1+9659+c1901784.x86_64
11:51:35      path: /usr/bin/conmon
11:51:35      version: 'conmon version 2.0.22, commit: 3ef8df793032fc21d5db1f32c0b3e5a73998381c'
11:51:35    cpus: 2
11:51:35    distribution:
11:51:35      distribution: '"ol"'
11:51:35      version: "8.3"
11:51:35    eventLogger: file
11:51:35    hostname: pasnlsdleng01.amer.dell.com
11:51:35    idMappings:
11:51:35      gidmap:
11:51:35      - container_id: 0
11:51:35        host_id: 436
11:51:35        size: 1
11:51:35      - container_id: 1
11:51:35        host_id: 100000
11:51:35        size: 65536
11:51:35      uidmap:
11:51:35      - container_id: 0
11:51:35        host_id: 436
11:51:35        size: 1
11:51:35      - container_id: 1
11:51:35        host_id: 100000
11:51:35        size: 65536
11:51:35    kernel: 4.18.0-193.14.3.el8_2.x86_64
11:51:35    linkmode: dynamic
11:51:35    memFree: 11187240960
11:51:35    memTotal: 16644784128
11:51:35    ociRuntime:
11:51:35      name: runc
11:51:35      package: runc-1.0.0-70.rc92.module+el8.3.1+9659+c1901784.x86_64
11:51:35      path: /usr/bin/runc
11:51:35      version: 'runc version spec: 1.0.2-dev'
11:51:35    os: linux
11:51:35    remoteSocket:
11:51:35      path: /run/user/436/podman/podman.sock
11:51:35    rootless: true
11:51:35    slirp4netns:
11:51:35      executable: /bin/slirp4netns
11:51:35      package: slirp4netns-1.1.8-1.module+el8.3.1+9659+c1901784.x86_64
11:51:35      version: |-
11:51:35        slirp4netns version 1.1.8
11:51:35        commit: d361001f495417b880f20329121e3aa431a8f90f
11:51:35        libslirp: 4.3.1
11:51:35        SLIRP_CONFIG_VERSION_MAX: 3
11:51:35        libseccomp: 2.4.3
11:51:35    swapFree: 8589918208
11:51:35    swapTotal: 8590979072
11:51:35    uptime: 5558h 56m 34.87s (Approximately 231.58 days)
11:51:35  registries:
11:51:35    search:
11:51:35    - container-registry.oracle.com
11:51:35    - docker.io
11:51:35    - registry.fedoraproject.org
11:51:35    - quay.io
11:51:35    - registry.centos.org
11:51:35  store:
11:51:35    configFile: /var/lib/jenkins/.config/containers/storage.conf
11:51:35    containerStore:
11:51:35      number: 0
11:51:35      paused: 0
11:51:35      running: 0
11:51:35      stopped: 0
11:51:35    graphDriverName: overlay
11:51:35    graphOptions:
11:51:35      overlay.mount_program:
11:51:35        Executable: /bin/fuse-overlayfs
11:51:35        Package: fuse-overlayfs-1.3.0-2.module+el8.3.1+9659+c1901784.x86_64
11:51:35        Version: |-
11:51:35          fusermount3 version: 3.2.1
11:51:35          fuse-overlayfs: version 1.3
11:51:35          FUSE library version 3.2.1
11:51:35          using FUSE kernel interface version 7.26
11:51:35    graphRoot: /var/lib/jenkins/.local/share/containers/storage
11:51:35    graphStatus:
11:51:35      Backing Filesystem: xfs
11:51:35      Native Overlay Diff: "false"
11:51:35      Supports d_type: "true"
11:51:35      Using metacopy: "false"
11:51:35    imageStore:
11:51:35      number: 81
11:51:35    runRoot: /run/user/436/containers
11:51:35    volumePath: /var/lib/jenkins/.local/share/containers/storage/volumes
11:51:35  version:
11:51:35    APIVersion: "2"
11:51:35    Built: 1613735160
11:51:35    BuiltTime: Fri Feb 19 06:46:00 2021
11:51:35    GitCommit: ""
11:51:35    GoVersion: go1.14.12
11:51:35    OsArch: linux/amd64
11:51:35    Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman): podman-3.1.0-1.fc34.x86_64.rpm

Output error :

time="2021-05-19T02:27:23-04:00" level=error msg="error cleaning up container 9547f93a15b4df3104b4192d15bbc6b2097c4acbc32d092a434c29878832c7b4: invalid reference format"

Error: error creating image volume "/var/lib/jenkins/.local/share/containers/storage/overlay-images/0bfbbb5fe1dba30662287087ac0af3093020d521e50a1e6afedc8d54c306b4b0":"/opt/pas-goat-appsec": invalid reference format

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? Tested with podman-3.1.0-1.fc34.x86_64.rpm

Additional environment details (AWS, VirtualBox, physical, etc.): Jenkins pipeline (Oracle Linux Server 8.3)

Saini01Basu commented 3 years ago

Have also tried -

podman run --privileged --rm -t --mount type=image,source=pas-goat-appsec:jenkins,destination=/opt/pas-goat-appsec:jenkins,rw=true -v /var/lib/jenkins/.local/share/containers/:/var/lib/containers/ harbor.****.com/*********/scan-automation:ee6898aa /bin/bash -c 'twistcli images scan --address https://containersecurity.****.com/ --user ************** --password *************** --project PKS --details pas-goat-appsec:jenkins'

On running above i got following error :

Error: Could not get runtime: database libpod temporary files directory (tmpdir) "/run/user/436/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/var/run/libpod": database configuration mismatch

Saini01Basu commented 3 years ago

Also did a run without mounting image volume and directly giving image repository path -

podman run --privileged --rm -t -v /var/lib/jenkins/.local/share/containers/:/var/lib/containers/ harbor.****.com/*********/scan-automation:ee6898aa /bin/bash -c 'twistcli images scan --address https://containersecurity.****.com/ --user **************** --password ***************** --project PKS --details --podman-path podman harbor.****.com/pasgoat/pas-goat-appsec:jenkins'

Got the same error as above: Error: Could not get runtime: database libpod temporary files directory (tmpdir) "/run/user/436/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/var/run/libpod": database configuration mismatch

vrothberg commented 3 years ago

Thanks for reaching out!

The problem is that /var/lib/jenkins/.local/share/containers/storage/overlay-images/0bfbbb5fe1dba30662287087ac0af3093020d521e50a1e6afedc8d54c306b4b0 is not a valid image reference. The source of an image mount is not meant to be a path on the host but a reference to an image as, for instance, in podman run $image.

It should work when use the name or ID of the image.

Saini01Basu commented 3 years ago

@vrothberg I have tried with image name as well - https://github.com/containers/podman/issues/10397#issuecomment-843803366 Got a different error : Error: Could not get runtime: database libpod temporary files directory (tmpdir) "/run/user/436/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/var/run/libpod": database configuration mismatch

vrothberg commented 3 years ago

Got a different error : Error: Could not get runtime: database libpod temporary files directory (tmpdir) "/run/user/436/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/var/run/libpod": database configuration mismatch

That looks like the tmpdir of root and rootless were mixed. Do other commands work (e.g., podman run --rm busybox ls)? I assume you're running as a rootless user. Can you have a look at ~/.config/containers/storage.conf and see if there's something changing the tmpdir?

Saini01Basu commented 3 years ago

@vrothberg

The container that I am running has both docker and podman installed in it as of now, the host where I am running the container has only podman installed. Is this causing the issue? Yes m running the container as a rootless user on the host

vrothberg commented 3 years ago

Docker and Podman can live happily side-by-side on the same machine.

Can you check the questions below?

Do other commands work (e.g., podman run --rm busybox ls)? I assume you're running as a rootless user. Can you have a look at ~/.config/containers/storage.conf and see if there's something changing the tmpdir?

Saini01Basu commented 3 years ago 
13:06:39  + podman run --rm busybox ls
13:06:39  Completed short name "busybox" with unqualified-search registries (origin: /etc/containers/registries.conf)
13:06:39  Trying to pull container-registry.oracle.com/busybox:latest...
13:06:41    unable to retrieve auth token: invalid username/password: unauthorized: authentication required
13:06:41  Trying to pull docker.io/library/busybox:latest...
13:09:18    toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
13:09:18  Trying to pull registry.fedoraproject.org/busybox:latest...
13:09:18    manifest unknown: manifest unknown
13:09:18  Trying to pull quay.io/busybox:latest...
13:09:18    StatusCode: 404, <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final/...
13:09:18  Trying to pull registry.centos.org/busybox:latest...
13:09:18    Get "https://registry.centos.org/v2/": x509: certificate signed by unknown authority
13:09:18  Error: 5 errors occurred while pulling:
13:09:18   * Error initializing source docker://container-registry.oracle.com/busybox:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required
13:09:18   * Error determining manifest MIME type for docker://busybox:latest: Error reading manifest sha256:f3cfc9d0dbf931d3db4685ec659b7ac68e2a578219da4aae65427886e649b06b in docker.io/library/busybox: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
13:09:18   * Error initializing source docker://registry.fedoraproject.org/busybox:latest: Error reading manifest latest in registry.fedoraproject.org/busybox: manifest unknown: manifest unknown
13:09:18   * Error initializing source docker://quay.io/busybox:latest: Error reading manifest latest in quay.io/busybox: StatusCode: 404, <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final/...
13:09:18   * Error initializing source docker://registry.centos.org/busybox:latest: error pinging docker registry registry.centos.org: Get "https://registry.centos.org/v2/": x509: certificate signed by unknown authority

When I tried to look into storage.conf very weirdly i get - cat: /var/lib/jenkins/.config/containers/storage.conf: No such file or directory

However podman build, podman pull and even podman run for example -

podman run --rm -t -v ${WORKSPACE}:/var/lib/jenkins/workspace/Jenkins-Pasgoat-Pipeline -w /var/lib/jenkins/workspace/Jenkins-Pasgoat-Pipeline harbor.****.com/*********/scan-automation:ee6898aa /bin/bash -c 'dastctl $AppScanSERVER $USERNAME $PASSWORD $FIID'

work as expected

vrothberg commented 3 years ago

Looks like you hit the Docker Hub rate limit: can you try with podman run --rm fedora ls?

Saini01Basu commented 3 years ago

podman run --rm fedora ls

This worked -

13:21:59  + podman run --rm fedora ls
13:21:59  Completed short name "fedora" with unqualified-search registries (origin: /etc/containers/registries.conf)
13:21:59  Trying to pull container-registry.oracle.com/fedora:latest...
13:22:02    unable to retrieve auth token: invalid username/password: unauthorized: authentication required
13:22:02  Trying to pull docker.io/library/fedora:latest...
13:24:23    toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
13:24:23  Trying to pull registry.fedoraproject.org/fedora:latest...
13:24:23  Getting image source signatures
13:24:23  Copying blob sha256:7679c09af3851a1622782c74864351c296a0d1886813862fd7116383aeba9f07
13:24:23  Copying config sha256:3567369c671193f96f057f76e3e136ecbd3fdc7065019cc3dd6ed5a96894f128
13:24:23  Writing manifest to image destination
13:24:23  Storing signatures
13:24:23  bin
13:24:23  boot
13:24:23  dev
13:24:23  etc
13:24:23  home
13:24:23  lib
13:24:23  lib64
13:24:23  lost+found
13:24:23  media
13:24:23  mnt
13:24:23  opt
13:24:23  proc
13:24:23  root
13:24:23  run
13:24:23  sbin
13:24:23  srv
13:24:23  sys
13:24:23  tmp
13:24:23  usr
13:24:23  var
vrothberg commented 3 years ago

Could you try podman system reset? Note that this will remove all Podman data and set it back to defaults.

Saini01Basu commented 3 years ago

If you notice m trying mount a volume as well in the command like - 

-v /var/lib/jenkins/.local/share/containers/:/var/lib/containers/

database configuration mismatch may be between the host and the inside the container that m running. m running script inside the container if u notice that starts with twistcli that uses podman podman system reset didn't help still get same issue

Saini01Basu commented 3 years ago

Also after doing a podman system reset in jenkins pipeline m getting error for all podman commands - Error: error creating tmpdir: mkdir /run/user/436: permission denied :(

vrothberg commented 3 years ago

Ah, you're running podman inside another container? There are a lot of requirements to get that running.

Are you running inside a privileged container?

@rhatdan PTAL

Saini01Basu commented 3 years ago

podman run --privileged --rm -t -v /var/lib/jenkins/.local/share/containers/:/var/lib/containers/ harbor.^^^^.com/*/scan-automation:ee6898aa /bin/bash -c 'twistcli images scan --address https://containersecurity.****.com/ --user **** --password ***** --project PKS --details --podman-path podman harbor.^^^^.com/pasgoat/pas-goat-appsec:jenkins'

I have highlighted options in above command which is relevant to this issue

rhatdan commented 3 years ago

Don't do the system reset, it must be attempting to remove and recreate the /run/user/436 directory which a non root user would not be allowed to do.

rhatdan commented 3 years ago

It looks to me like everything is working. RUnning podman system reset within a privileged container si going to attempt to remove the images/containers that are shared into the container and the running container itself.

Saini01Basu commented 3 years ago

podman run --privileged --rm -t -v /var/lib/jenkins/.local/share/containers/:/var/lib/containers/ harbor.^^^^.com/*/scan-automation:ee6898aa /bin/bash -c 'twistcli images scan --address https://containersecurity.****.com/ --user **** --password ***** --project PKS --details --podman-path podman harbor.^^^^.com/pasgoat/pas-goat-appsec:jenkins'

I have highlighted options in above command which is relevant to this issue

I was getting the following error :

Error: Could not get runtime: database libpod temporary files directory (tmpdir) "/run/user/436/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/var/run/libpod": database configuration mismatch
Saini01Basu commented 3 years ago

So i was able to fix podman commands post the podman system reset just restarted the jenkins service. Can you guys give me the list of requirements for running podman inside container rootless mode? @rhatdan @vrothberg

It's like i want to be able to do podman pull <> inside another container (which has podman installed)

Saini01Basu commented 3 years ago

Was able to make it work with below command -

podman run --privileged --rm -t --volume=/var/lib/jenkins/.local/share/containers/storage:/var/lib/containers/storage harbor.^^^^^.com/*/scan-automation:ee6898aa /bin/bash -c 'twistcli images scan --address https://containersecurity.^^^^^.com/ --user ***** --password *** --project PKS --containerized --details --output-file results.json harbor.^^^^^.com/pasgoat/pas-goat-appsec:jenkins && cat results.json | jq -rC ".[]"'

The highlighted options in the command did the trick. Anyway Thanks guys

rhatdan commented 3 years ago

No.