search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.54k stars 2.39k forks source link

/var/lib/containers/storage/overlay/(...)/merged/bin/bash: no such file or directory #2834

Closed Noed closed 5 years ago

Noed commented 5 years ago

/kind bug

Description When pushing an image, podman try to get /bin/bash from a container storage directory but the file doesn't exist.

Steps to reproduce the issue:

  1. docker run --privileged --name fedora --rm -i -t fedora:latest /bin/bash
  2. yum -y install podman
  3. sed -i 's/#mount_program/mount_program/g' /etc/containers/storage.conf
  4. podman build -t ng-hello .
  5. podman tag ng-hello us.icr.io/chaval-namespace/ng-hello
  6. podman push us.icr.io/chaval-namespace/ng-hello

Describe the results you received:

[root@910318e6d309 ng-hello]# podman push us.icr.io/chaval-namespace/ng-hello
Getting image source signatures
Copying blob fbb641a8b943 [======================================] 100.7MiB / 100.7MiB
Copying blob 604829a174eb [======================================] 23.0MiB / 23.0MiB
Copying blob 12cb127eee44 [======================================] 7.6MiB / 7.6MiB
Copying blob b17cc31e431b [======================================] 139.6MiB / 139.6MiB
Copying blob 0fe19df8b8f8 [======================================] 549.7MiB / 549.7MiB
Copying blob 0095ff73bb21 [======================================] 341.0KiB / 341.0KiB
Copying blob d0a65b1da309 [======================================] 63.8MiB / 63.8MiB
Copying blob e4396ca14c33 [======================================] 4.9MiB / 4.9MiB
Copying blob f8af2b549ec9 [======================================] 369.5KiB / 369.5KiB
Copying blob 5f70bf18a086 [======================================] 1.0KiB / 1.0KiB
Copying blob 195c27707b98 [======================================] 623.5KiB / 623.5KiB
Copying blob 5f70bf18a086 [======================================] 1.0KiB / 1.0KiB
Copying blob 5f70bf18a086 [======================================] 1.0KiB / 1.0KiB
Error: Error copying image to the remote destination: Error writing blob: Patch https://us.icr.io/v2/chaval-namespace/ng-hello/blobs/uploads/a1e18d4c-e196-42a9-830b-20f41e3addf4?_state=rlg0WXzgv5y_wvZ9KsJsBEQU3ZMhY0mTIOVSLlKZH6p7Ik5hbWUiOiJjaGF2YWwtbmFtZXNwYWNlL25nLWhlbGxvIiwiVVVJRCI6ImExZTE4ZDRjLWUxOTYtNDJhOS04MzBiLTIwZjQxZTNhZGRmNCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxOS0wNC0wMlQxODo0MDoxMy44MTkzMjIxNDhaIn0%3D: open /var/lib/containers/storage/overlay/fbb641a8b94349e89886f65d79928e4673530e2a2b4d33c2c95e0426713f78e4/merged/bin/bash: no such file or directory

Describe the results you expected: If I do the same with docker, the push works fine.

Additional information you deem important (e.g. issue happens only occasionally): The /bin/bash is not where podman is trying to read

[root@910318e6d309 ng-hello]# ls -laR /var/lib/containers/storage/overlay/fbb641a8b94349e89886f65d79928e4673530e2a2b4d33c2c95e0426713f78e4/merged/
/var/lib/containers/storage/overlay/fbb641a8b94349e89886f65d79928e4673530e2a2b4d33c2c95e0426713f78e4/merged/:
total 8
drwx------ 2 root root 4096 Apr  2 17:54 .
drwx------ 6 root root 4096 Apr  2 17:54 ..

I'm pushing to a private repository on IBM Cloud.

Output of podman version:

Version:            1.1.2
RemoteAPI Version:  1
Go Version:         go1.11.5
Git Commit:         a95a49d3038462d033f84ac314ec8a3064a99cff
Built:              Tue Mar  5 18:10:31 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: a95a49d3038462d033f84ac314ec8a3064a99cff
  go version: go1.11.5
  podman version: 1.1.2
host:
  BuildahVersion: 1.7.1
  Conmon:
    package: podman-1.1.2-1.git0ad9b6b.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: a95a49d3038462d033f84ac314ec8a3064a99cff'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 193257472
  MemTotal: 2076532736
  OCIRuntime:
    package: runc-1.0.0-68.dev.git6635b4f.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: ef9132178ccc3d2775d4fb51f1e431f30cac1398-dirty
      spec: 1.0.1-dev
  SwapFree: 1052901376
  SwapTotal: 1073737728
  arch: amd64
  cpus: 2
  hostname: 910318e6d309
  kernel: 4.9.125-linuxkit
  os: linux
  rootless: false
  uptime: 6h 20m 8.56s (Approximately 0.25 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  - overlay.mountopt=nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 6
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Dockerfile:

FROM node:latest
COPY package*.json ./
CMD npm install
COPY . .
EXPOSE 4200
CMD npm start

This is an Angular quickstart application running on node.js

IBM Cloud CLI installation:

curl -fsSL https://clis.ng.bluemix.net/install/linux | sh
ibmcloud login --apikey ****** -a api.ng.bluemix.net
ibmcloud plugin install container-registry
ln -s /usr/bin/podman /usr/bin/docker
ibmcloud cr login
ibmcloud cr namespaces

This will write the correct secrets to /run/containers/0/auth.json

This simple one works:

[root@910318e6d309 ~]# cat Dockerfile
FROM scratch
CMD echo Hello

[root@910318e6d309 ~]# podman build -t hello .
STEP 1: FROM scratch
STEP 2: CMD echo Hello
--> 7381a247a0dbce5ac13216879bfd840b677c5d9d5fa99d896ada2dd2b2e8df12
STEP 3: COMMIT hello

[root@910318e6d309 ~]# podman tag hello us.icr.io/chaval-namespace/hello

[root@910318e6d309 ~]# podman push us.icr.io/chaval-namespace/hello
Getting image source signatures
Copying blob 5f70bf18a086 [======================================] 1.0KiB / 1.0KiB
Copying config 7381a247a0 [======================================] 418b / 418b
Writing manifest to image destination
Copying config 7381a247a0 [======================================] 418b / 418b
Writing manifest to image destination
Storing signatures

[root@910318e6d309 ~]# ibmcloud cr images
Listing images...

REPOSITORY                         TAG      DIGEST         NAMESPACE          CREATED         SIZE   SECURITY STATUS
us.icr.io/chaval-namespace/hello   latest   0317e4005837   chaval-namespace   3 minutes ago   48 B   No Issues

OK
mheon commented 5 years ago

To make sure I understand: You're running Podman in a Docker container, using Podman build, and getting an error pushing to a repository?

I don't believe we've ever tried this without a privileged container on the outside.

@giuseppe It looks like fuse-overlay is in use here... Could that be it?

Noed commented 5 years ago

Yes, the container is fedora:latest started privileged with Docker:

docker run --privileged --name fedora --rm -i -t fedora:latest /bin/bash

And yes I'm using fuse-overlayfs:

[root@910318e6d309 ~]# grep fuse-overlayfs /etc/containers/storage.conf
mount_program = "/usr/bin/fuse-overlayfs"

Then I install podman on it, checkout code from git, build an image, tag and try to push to a repository. This is because I will run in a Jenkins CI/CD pipeline that runs the agents in containers. The Jenkins script stops at the same point.

Here's the full Jenkins build output:

Started by user admin
Obtained Jenkinsfile from git https://github.com/Noed/ng-hello
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
‘mypod-402p0-v5lcf’ is offline
Agent mypod-402p0-v5lcf is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (mypod): 
* [podman] fedora:latest

Running on mypod-402p0-v5lcf in /home/jenkins/workspace/ng-hello-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] checkout
using credential github.com
Cloning the remote Git repository
Cloning repository https://github.com/Noed/ng-hello
 > git init /home/jenkins/workspace/ng-hello-pipeline # timeout=10
Fetching upstream changes from https://github.com/Noed/ng-hello
 > git --version # timeout=10
using GIT_ASKPASS to set credentials github.com
 > git fetch --tags --force --progress https://github.com/Noed/ng-hello +refs/heads/*:refs/remotes/origin/*
Checking out Revision 9ac63d407a5b540f22c42a8cc3c635739f2b059c (refs/remotes/origin/master)
 > git config remote.origin.url https://github.com/Noed/ng-hello # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/Noed/ng-hello # timeout=10
Fetching upstream changes from https://github.com/Noed/ng-hello
using GIT_ASKPASS to set credentials github.com
 > git fetch --tags --force --progress https://github.com/Noed/ng-hello +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9ac63d407a5b540f22c42a8cc3c635739f2b059c
Commit message: "minimal"
[Pipeline] }
[Pipeline] // stage
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Container info)
[Pipeline] sh
 > git rev-list --no-walk 9af915275bb54079195d082da9991c3965be6f01 # timeout=10
+ whoami
root
+ pwd
/home/jenkins/workspace/ng-hello-pipeline
+ cat /etc/os-release
NAME=Fedora
VERSION="29 (Container Image)"
ID=fedora
VERSION_ID=29
VERSION_CODENAME=""
PLATFORM_ID="platform:f29"
PRETTY_NAME="Fedora 29 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:29"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=29
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=29
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
+ uname -a
Linux mypod-402p0-v5lcf 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
+ df -BM
Filesystem     1M-blocks   Used Available Use% Mounted on
overlay           98956M 14636M    84304M  15% /
tmpfs                64M     0M       64M   0% /dev
tmpfs              1974M     0M     1974M   0% /sys/fs/cgroup
/dev/xvda2        98956M 14636M    84304M  15% /etc/hosts
shm                  64M     0M       64M   0% /dev/shm
tmpfs              1974M     1M     1974M   1% /run/secrets/kubernetes.io/serviceaccount
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Setup podman)
[Pipeline] sh
+ yum -y install podman
Fedora Modular 29 - x86_64                      362 kB/s | 1.5 MB     00:04    
Fedora Modular 29 - x86_64 - Updates            470 kB/s | 2.1 MB     00:04    
Fedora 29 - x86_64 - Updates                    5.2 MB/s |  25 MB     00:04    
Fedora 29 - x86_64                               12 MB/s |  62 MB     00:05    
Last metadata expiration check: 0:00:01 ago on Tue Apr  2 19:14:07 2019.
Dependencies resolved.
================================================================================
 Package                    Arch   Version                        Repo     Size
================================================================================
Installing:
 podman                     x86_64 1:1.1.2-1.git0ad9b6b.fc29      updates 9.6 M
Installing dependencies:
 container-selinux          noarch 2:2.91-1.gitacc6941.fc29       updates  46 k
 containernetworking-plugins
                            x86_64 0.7.4-2.fc29                   updates  13 M
 containers-common          x86_64 1:0.1.35-1.git404c5bd.fc29     updates  31 k
 fuse3-libs                 x86_64 3.4.2-1.fc29                   updates  82 k
 libnetfilter_conntrack     x86_64 1.0.7-1.fc29                   updates  56 k
 libselinux-utils           x86_64 2.8-6.fc29                     updates  99 k
 ostree-libs                x86_64 2019.1-3.fc29                  updates 363 k
 policycoreutils            x86_64 2.8-17.fc29                    updates 187 k
 runc                       x86_64 2:1.0.0-68.dev.git6635b4f.fc29 updates 2.3 M
 selinux-policy             noarch 3.14.2-51.fc29                 updates 118 k
 selinux-policy-targeted    noarch 3.14.2-51.fc29                 updates  14 M
 iptables                   x86_64 1.8.0-3.fc29                   fedora  478 k
 jansson                    x86_64 2.11-2.fc29                    fedora   43 k
 kmod                       x86_64 25-3.fc29                      fedora  112 k
 libmnl                     x86_64 1.0.4-8.fc29                   fedora   27 k
 libnet                     x86_64 1.1.6-16.fc29                  fedora   62 k
 libnfnetlink               x86_64 1.0.1-14.fc29                  fedora   29 k
 libnftnl                   x86_64 1.1.1-5.fc29                   fedora   88 k
 libnl3                     x86_64 3.4.0-6.fc29                   fedora  304 k
 nftables                   x86_64 1:0.9.0-2.fc29                 fedora  225 k
 protobuf-c                 x86_64 1.3.0-5.fc29                   fedora   33 k
Installing weak dependencies:
 criu                       x86_64 3.11-1.fc29                    updates 487 k
 fuse-overlayfs             x86_64 0.3-8.dev.gita6958ce.fc29      updates  49 k
 slirp4netns                x86_64 0.3-0.alpha.2.git30883b5.fc29  updates  71 k

Transaction Summary
================================================================================
Install  25 Packages

Total download size: 42 M
Installed size: 151 M
Downloading Packages:
(1/25): containers-common-0.1.35-1.git404c5bd.f  23 kB/s |  31 kB     00:01    
(2/25): container-selinux-2.91-1.gitacc6941.fc2  34 kB/s |  46 kB     00:01    
(3/25): fuse-overlayfs-0.3-8.dev.gita6958ce.fc2 432 kB/s |  49 kB     00:00    
(4/25): fuse3-libs-3.4.2-1.fc29.x86_64.rpm      716 kB/s |  82 kB     00:00    
(5/25): criu-3.11-1.fc29.x86_64.rpm             1.8 MB/s | 487 kB     00:00    
(6/25): libnetfilter_conntrack-1.0.7-1.fc29.x86 852 kB/s |  56 kB     00:00    
(7/25): libselinux-utils-2.8-6.fc29.x86_64.rpm  1.2 MB/s |  99 kB     00:00    
(8/25): ostree-libs-2019.1-3.fc29.x86_64.rpm    2.2 MB/s | 363 kB     00:00    
(9/25): containernetworking-plugins-0.7.4-2.fc2 6.8 MB/s |  13 MB     00:01    
(10/25): policycoreutils-2.8-17.fc29.x86_64.rpm 1.3 MB/s | 187 kB     00:00    
(11/25): selinux-policy-3.14.2-51.fc29.noarch.r 1.7 MB/s | 118 kB     00:00    
(12/25): runc-1.0.0-68.dev.git6635b4f.fc29.x86_  14 MB/s | 2.3 MB     00:00    
(13/25): podman-1.1.2-1.git0ad9b6b.fc29.x86_64.  18 MB/s | 9.6 MB     00:00    
(14/25): slirp4netns-0.3-0.alpha.2.git30883b5.f 660 kB/s |  71 kB     00:00    
(15/25): jansson-2.11-2.fc29.x86_64.rpm         157 kB/s |  43 kB     00:00    
(16/25): selinux-policy-targeted-3.14.2-51.fc29  24 MB/s |  14 MB     00:00    
(17/25): kmod-25-3.fc29.x86_64.rpm              904 kB/s | 112 kB     00:00    
(18/25): iptables-1.8.0-3.fc29.x86_64.rpm       1.0 MB/s | 478 kB     00:00    
(19/25): libnet-1.1.6-16.fc29.x86_64.rpm        700 kB/s |  62 kB     00:00    
(20/25): libnfnetlink-1.0.1-14.fc29.x86_64.rpm  460 kB/s |  29 kB     00:00    
(21/25): libmnl-1.0.4-8.fc29.x86_64.rpm         209 kB/s |  27 kB     00:00    
(22/25): libnftnl-1.1.1-5.fc29.x86_64.rpm       1.0 MB/s |  88 kB     00:00    
(23/25): libnl3-3.4.0-6.fc29.x86_64.rpm         4.1 MB/s | 304 kB     00:00    
(24/25): protobuf-c-1.3.0-5.fc29.x86_64.rpm     463 kB/s |  33 kB     00:00    
(25/25): nftables-0.9.0-2.fc29.x86_64.rpm       1.2 MB/s | 225 kB     00:00    
--------------------------------------------------------------------------------
Total                                           7.1 MB/s |  42 MB     00:06     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : libmnl-1.0.4-8.fc29.x86_64                            1/25 
  Running scriptlet: libmnl-1.0.4-8.fc29.x86_64                            1/25 
  Installing       : libnfnetlink-1.0.1-14.fc29.x86_64                     2/25 
  Running scriptlet: libnfnetlink-1.0.1-14.fc29.x86_64                     2/25 
  Installing       : libselinux-utils-2.8-6.fc29.x86_64                    3/25 
  Installing       : policycoreutils-2.8-17.fc29.x86_64                    4/25 
  Running scriptlet: policycoreutils-2.8-17.fc29.x86_64                    4/25 
  Installing       : selinux-policy-3.14.2-51.fc29.noarch                  5/25 
  Running scriptlet: selinux-policy-3.14.2-51.fc29.noarch                  5/25 
  Running scriptlet: selinux-policy-targeted-3.14.2-51.fc29.noarch         6/25 
  Installing       : selinux-policy-targeted-3.14.2-51.fc29.noarch         6/25 
  Running scriptlet: selinux-policy-targeted-3.14.2-51.fc29.noarch         6/25 
  Installing       : container-selinux-2:2.91-1.gitacc6941.fc29.noarch     7/25 
  Running scriptlet: container-selinux-2:2.91-1.gitacc6941.fc29.noarch     7/25 
  Installing       : libnetfilter_conntrack-1.0.7-1.fc29.x86_64            8/25 
  Running scriptlet: libnetfilter_conntrack-1.0.7-1.fc29.x86_64            8/25 
  Installing       : iptables-1.8.0-3.fc29.x86_64                          9/25 
  Running scriptlet: iptables-1.8.0-3.fc29.x86_64                          9/25 
  Installing       : protobuf-c-1.3.0-5.fc29.x86_64                       10/25 
  Installing       : libnl3-3.4.0-6.fc29.x86_64                           11/25 
  Running scriptlet: libnl3-3.4.0-6.fc29.x86_64                           11/25 
  Installing       : libnet-1.1.6-16.fc29.x86_64                          12/25 
  Running scriptlet: libnet-1.1.6-16.fc29.x86_64                          12/25 
  Installing       : criu-3.11-1.fc29.x86_64                              13/25 
  Installing       : runc-2:1.0.0-68.dev.git6635b4f.fc29.x86_64           14/25 
  Installing       : kmod-25-3.fc29.x86_64                                15/25 
  Installing       : jansson-2.11-2.fc29.x86_64                           16/25 
  Installing       : libnftnl-1.1.1-5.fc29.x86_64                         17/25 
  Installing       : nftables-1:0.9.0-2.fc29.x86_64                       18/25 
  Running scriptlet: nftables-1:0.9.0-2.fc29.x86_64                       18/25 
  Installing       : slirp4netns-0.3-0.alpha.2.git30883b5.fc29.x86_64     19/25 
  Installing       : ostree-libs-2019.1-3.fc29.x86_64                     20/25 
  Installing       : fuse3-libs-3.4.2-1.fc29.x86_64                       21/25 
  Installing       : fuse-overlayfs-0.3-8.dev.gita6958ce.fc29.x86_64      22/25 
  Running scriptlet: fuse-overlayfs-0.3-8.dev.gita6958ce.fc29.x86_64      22/25 
  Installing       : containers-common-1:0.1.35-1.git404c5bd.fc29.x86_6   23/25 
  Installing       : containernetworking-plugins-0.7.4-2.fc29.x86_64      24/25 
  Installing       : podman-1:1.1.2-1.git0ad9b6b.fc29.x86_64              25/25 
  Running scriptlet: podman-1:1.1.2-1.git0ad9b6b.fc29.x86_64              25/25 
  Verifying        : container-selinux-2:2.91-1.gitacc6941.fc29.noarch     1/25 
  Verifying        : containernetworking-plugins-0.7.4-2.fc29.x86_64       2/25 
  Verifying        : containers-common-1:0.1.35-1.git404c5bd.fc29.x86_6    3/25 
  Verifying        : criu-3.11-1.fc29.x86_64                               4/25 
  Verifying        : fuse-overlayfs-0.3-8.dev.gita6958ce.fc29.x86_64       5/25 
  Verifying        : fuse3-libs-3.4.2-1.fc29.x86_64                        6/25 
  Verifying        : libnetfilter_conntrack-1.0.7-1.fc29.x86_64            7/25 
  Verifying        : libselinux-utils-2.8-6.fc29.x86_64                    8/25 
  Verifying        : ostree-libs-2019.1-3.fc29.x86_64                      9/25 
  Verifying        : podman-1:1.1.2-1.git0ad9b6b.fc29.x86_64              10/25 
  Verifying        : policycoreutils-2.8-17.fc29.x86_64                   11/25 
  Verifying        : runc-2:1.0.0-68.dev.git6635b4f.fc29.x86_64           12/25 
  Verifying        : selinux-policy-3.14.2-51.fc29.noarch                 13/25 
  Verifying        : selinux-policy-targeted-3.14.2-51.fc29.noarch        14/25 
  Verifying        : slirp4netns-0.3-0.alpha.2.git30883b5.fc29.x86_64     15/25 
  Verifying        : iptables-1.8.0-3.fc29.x86_64                         16/25 
  Verifying        : jansson-2.11-2.fc29.x86_64                           17/25 
  Verifying        : kmod-25-3.fc29.x86_64                                18/25 
  Verifying        : libmnl-1.0.4-8.fc29.x86_64                           19/25 
  Verifying        : libnet-1.1.6-16.fc29.x86_64                          20/25 
  Verifying        : libnfnetlink-1.0.1-14.fc29.x86_64                    21/25 
  Verifying        : libnftnl-1.1.1-5.fc29.x86_64                         22/25 
  Verifying        : libnl3-3.4.0-6.fc29.x86_64                           23/25 
  Verifying        : nftables-1:0.9.0-2.fc29.x86_64                       24/25 
  Verifying        : protobuf-c-1.3.0-5.fc29.x86_64                       25/25 

Installed:
  podman-1:1.1.2-1.git0ad9b6b.fc29.x86_64                                       
  criu-3.11-1.fc29.x86_64                                                       
  fuse-overlayfs-0.3-8.dev.gita6958ce.fc29.x86_64                               
  slirp4netns-0.3-0.alpha.2.git30883b5.fc29.x86_64                              
  container-selinux-2:2.91-1.gitacc6941.fc29.noarch                             
  containernetworking-plugins-0.7.4-2.fc29.x86_64                               
  containers-common-1:0.1.35-1.git404c5bd.fc29.x86_64                           
  fuse3-libs-3.4.2-1.fc29.x86_64                                                
  libnetfilter_conntrack-1.0.7-1.fc29.x86_64                                    
  libselinux-utils-2.8-6.fc29.x86_64                                            
  ostree-libs-2019.1-3.fc29.x86_64                                              
  policycoreutils-2.8-17.fc29.x86_64                                            
  runc-2:1.0.0-68.dev.git6635b4f.fc29.x86_64                                    
  selinux-policy-3.14.2-51.fc29.noarch                                          
  selinux-policy-targeted-3.14.2-51.fc29.noarch                                 
  iptables-1.8.0-3.fc29.x86_64                                                  
  jansson-2.11-2.fc29.x86_64                                                    
  kmod-25-3.fc29.x86_64                                                         
  libmnl-1.0.4-8.fc29.x86_64                                                    
  libnet-1.1.6-16.fc29.x86_64                                                   
  libnfnetlink-1.0.1-14.fc29.x86_64                                             
  libnftnl-1.1.1-5.fc29.x86_64                                                  
  libnl3-3.4.0-6.fc29.x86_64                                                    
  nftables-1:0.9.0-2.fc29.x86_64                                                
  protobuf-c-1.3.0-5.fc29.x86_64                                                

Complete!
+ podman --version
podman version 1.1.2
+ sed -i s/#mount_program/mount_program/g /etc/containers/storage.conf
+ grep fuse-overlayfs /etc/containers/storage.conf
mount_program = "/usr/bin/fuse-overlayfs"
+ podman info
host:
  BuildahVersion: 1.7.1
  Conmon:
    package: podman-1.1.2-1.git0ad9b6b.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: a95a49d3038462d033f84ac314ec8a3064a99cff'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 1034637312
  MemTotal: 4138549248
  OCIRuntime:
    package: runc-1.0.0-68.dev.git6635b4f.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: ef9132178ccc3d2775d4fb51f1e431f30cac1398-dirty
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: mypod-402p0-v5lcf
  kernel: 4.4.0-142-generic
  os: linux
  rootless: false
  uptime: 621h 15m 29.92s (Approximately 25.88 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  - overlay.mountopt=nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Setup IBM Cloud CLI)
[Pipeline] withCredentials
Masking only exact matches of $APIKEY
[Pipeline] {
[Pipeline] sh
+ curl -fsSL https://clis.ng.bluemix.net/install/linux
+ sh
Current platform is linux64. Downloading corresponding IBM Cloud CLI...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   120    0   120    0     0   1348      0 --:--:-- --:--:-- --:--:--  1348
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 61 15.0M   61 9480k    0     0  7854k      0  0:00:01  0:00:01 --:--:-- 9358k
100 15.0M  100 15.0M    0     0  8358k      0  0:00:01  0:00:01 --:--:-- 9339k
Download complete. Executing installer...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    40    0    40    0     0    666      0 --:--:-- --:--:-- --:--:--   666
Bluemix_CLI/
Bluemix_CLI/autocomplete/
Bluemix_CLI/autocomplete/bash_autocomplete
Bluemix_CLI/autocomplete/zsh_autocomplete
Bluemix_CLI/install
Bluemix_CLI/install_bluemix_cli
Bluemix_CLI/bin/
Bluemix_CLI/bin/ibmcloud-analytics
Bluemix_CLI/bin/ibmcloud
Bluemix_CLI/bin/cfcli/
Bluemix_CLI/bin/cfcli/cf
Bluemix_CLI/bin/LICENSE
Bluemix_CLI/bin/NOTICE
Bluemix_CLI/uninstall
Copying files ...
The installation completed successfully.
To get started, open a terminal window and enter "ibmcloud help". Refer to "http://ibm.biz/cli-auto-completion" if you want to enable auto-completion for zsh or bash.

IBM Cloud CLI has a plug-in framework to extend its capability. To install the recommended plug-ins and dependencies, run the install script from "http://ibm.biz/install-idt". For additional plug-in details, see "http://ibm.biz/install-cli-plugin".

IBM Cloud CLI automatically collects data for usage analysis and user experience improvement. To disable the collecting, run "ibmcloud config --usage-stats-collect false"

To uninstall, run "/usr/local/ibmcloud/uninstall"
Install complete.
+ ibmcloud -v
ibmcloud version 0.15.0+1bfd374-2019-04-01T02:34:48+00:00
+ ibmcloud login --apikey **** -a api.ng.bluemix.net
API endpoint: api.ng.bluemix.net
Authenticating...
OK

Targeted account (35ef691a3f0b2866c1b4218917443263)

Targeted resource group default

API endpoint:      https://api.ng.bluemix.net   
Region:            us-south   
User:              
Account:           (35ef691a3f0b2866c1b4218917443263)   
Resource group:    default   
CF API endpoint:      
Org:                  
Space:                

Tip: If you are managing Cloud Foundry applications and services
- Use 'ibmcloud target --cf' to target Cloud Foundry org/space interactively, or use 'ibmcloud target --cf-api ENDPOINT -o ORG -s SPACE' to target the org/space.
- Use 'ibmcloud cf' if you want to run the Cloud Foundry CLI with current IBM Cloud CLI context.

+ ibmcloud plugin install container-registry
Looking up 'container-registry' from repository 'IBM Cloud'...
Plug-in 'container-registry 0.1.373' found in repository 'IBM Cloud'
Attempting to download the binary file...

 0 B / 20.23 MiB    0.00%
 672.00 KiB / 20.23 MiB    3.24% 5s
 4.05 MiB / 20.23 MiB   20.00% 1s
 6.80 MiB / 20.23 MiB   33.59% 1s
 11.21 MiB / 20.23 MiB   55.41% 0s
 15.59 MiB / 20.23 MiB   77.07% 0s
 18.79 MiB / 20.23 MiB   92.86% 0s
 20.23 MiB / 20.23 MiB  100.00% 1s
21217024 bytes downloaded
Installing binary...
OK
Plug-in 'container-registry 0.1.373' was successfully installed into /home/jenkins/.bluemix/plugins/container-registry. Use 'ibmcloud plugin show container-registry' to show its details.
+ ln -s /usr/bin/podman /usr/bin/docker
+ ibmcloud cr login
Logging in to 'registry.ng.bluemix.net'...
Logged in to 'registry.ng.bluemix.net'.

IBM Cloud Container Registry is adopting new icr.io domain names to align with the rebranding of IBM Cloud for a better user experience. The existing bluemix.net domain names are deprecated, but you can continue to use them for the time being, as an unsupported date will be announced later. For more information about registry domain names, see https://cloud.ibm.com/docs/services/Registry?topic=registry-registry_overview#registry_regions_local

Logging in to 'us.icr.io'...
Logged in to 'us.icr.io'.

IBM Cloud Container Registry is adopting new icr.io domain names to align with the rebranding of IBM Cloud for a better user experience. The existing bluemix.net domain names are deprecated, but you can continue to use them for the time being, as an unsupported date will be announced later. For more information about registry domain names, see https://cloud.ibm.com/docs/services/Registry?topic=registry-registry_overview#registry_regions_local

OK
+ ibmcloud cr namespaces
Listing namespaces...

Namespace   
chaval-namespace   

OK
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build application image)
[Pipeline] sh
+ podman build -t ng-hello .
STEP 1: FROM node:latest
Getting image source signatures
Copying blob sha256:e79bb959ec00faf01da52437df4fad4537ec669f60455a38ad583ec2b8f00498
Copying blob sha256:d4b7902036fe0cefdfe9ccf0404fe13322ecbd552f132be73d3e840f95538838
Copying blob sha256:1b2a72d4e03052566e99130108071fc4eca4942c62923e3e5cf19666a23088ef
Copying blob sha256:d54db43011fd116b8cb6d9e49e268cee1fa6212f152b30cbfa7f3c4c684427c3
Copying blob sha256:69d473365bb390367b7a54a3e890ca28c4640a56dfe4f53a0036130c964a1e52
Copying blob sha256:6e2490ee2dc80ccc1ffd79d0bd91f1b5cb3ad9ed8d8849058b82ab0e14de9c40
Copying blob sha256:374acee750f966da051f898e9c94b778e76709d59092674d960daf964dc0df6e
Copying blob sha256:57b95309019bb755e7b9f4bfb99b8a45be12d62dd07749a0423635af0014f9fd
Copying config sha256:a9c1445cbd528d08e428ba06faa3c56743724ef0db67f52b8cb8ddf81e6f7f49
Writing manifest to image destination
Storing signatures
STEP 2: COPY package*.json ./
--> 96a4a1f95579c5735d1f21f226b4c8beaf45bb9e326a8a3c165ebb4f25c28c1b
STEP 3: FROM 96a4a1f95579c5735d1f21f226b4c8beaf45bb9e326a8a3c165ebb4f25c28c1b
STEP 4: CMD npm install
--> 7e68db4ddd0ed9ce445c92dce19008f522bcdebefc5cf8ed01c6cbf06dc16b7f
STEP 5: FROM 7e68db4ddd0ed9ce445c92dce19008f522bcdebefc5cf8ed01c6cbf06dc16b7f
STEP 6: EXPOSE 4200
--> 94f474ab0860ac42ee653279f91ea18298baddd59d65868e08567e84a56784a3
STEP 7: FROM 94f474ab0860ac42ee653279f91ea18298baddd59d65868e08567e84a56784a3
STEP 8: CMD npm start
--> 8b84097401068a0f834199ee1e0541089187e11a6900483bf506fe9a48a13f7f
STEP 9: COMMIT ng-hello
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Push image to repository)
[Pipeline] sh
+ podman tag ng-hello us.icr.io/chaval-namespace/ng-hello
+ podman push us.icr.io/chaval-namespace/ng-hello
Getting image source signatures
Copying blob sha256:fbb641a8b94349e89886f65d79928e4673530e2a2b4d33c2c95e0426713f78e4
Copying blob sha256:604829a174eb966a2102e2e68c7669e1fe56721e8d7ea27f9a286aa33be8be20
Copying blob sha256:12cb127eee44270330891b1b610ce34e81f53a91a22e3a7f53f0632391d99892
Copying blob sha256:b17cc31e431beb2f39988dff23d04f85ba4b446fc0a13f304774852fa3c87d85
Copying blob sha256:0fe19df8b8f8eb5f545f50e11f958bf37d27ab7da5c260a9bd2bc0ff6fb760a4
Copying blob sha256:0095ff73bb219f8cc528e9ddee70c2f84839ce2f55093e87e8048c4c9f201d2d
Copying blob sha256:d0a65b1da30968e03a4cb1ac8f2d76c6656b78dd4f0a79a3a539d4041e998e6f
Copying blob sha256:e4396ca14c332c42630f88b21aa9688e0b10b10dd4a3028ce875279632434b1b
Copying blob sha256:0acc6f380a1636abfb105db04c59cd810095ae818ddcd22c13336b6be21c9c2a
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
Error: Error copying image to the remote destination: Error writing blob: Patch https://us.icr.io/v2/chaval-namespace/ng-hello/blobs/uploads/26db2fcc-8ee1-4c40-9b2f-1b8e704541f9?_state=guctTP_BSar5hoefXd9kDfkG8PiTt4rozrfZjdmEVv97Ik5hbWUiOiJjaGF2YWwtbmFtZXNwYWNlL25nLWhlbGxvIiwiVVVJRCI6IjI2ZGIyZmNjLThlZTEtNGM0MC05YjJmLTFiOGU3MDQ1NDFmOSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxOS0wNC0wMlQxOToxNjozNy41NTUyNjk2OTFaIn0%3D: open /var/lib/containers/storage/overlay/fbb641a8b94349e89886f65d79928e4673530e2a2b4d33c2c95e0426713f78e4/merged/bin/bash: no such file or directory
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code 125
Finished: FAILURE
giuseppe commented 5 years ago

you'll need to bind mount a volume for /var/lib/containers/storage. Overlay on top of overlay is not going to work

Noed commented 5 years ago

Thanks @giuseppe Bind during the build or the push? Any doc how I can do this?

giuseppe commented 5 years ago

@Noed bind to the docker command, so that /var/lib/containers/storage in the container is not on overlay

Noed commented 5 years ago

@giuseppe awesome it worked as simple as that:

docker run --privileged --volume /var/lib/containers/storage --name fedora --rm -i -t fedora:latest /bin/bash

And worked on the Jenkinsfile too with this:

volumes: [
    hostPathVolume(mountPath: '/var/lib/containers/storage', hostPath: '/var/lib/containers/storage')
],

Now my build completes:

+ podman push us.icr.io/chaval-namespace/ng-hello
Getting image source signatures
Copying blob sha256:fbb641a8b94349e89886f65d79928e4673530e2a2b4d33c2c95e0426713f78e4
Copying blob sha256:604829a174eb966a2102e2e68c7669e1fe56721e8d7ea27f9a286aa33be8be20
Copying blob sha256:12cb127eee44270330891b1b610ce34e81f53a91a22e3a7f53f0632391d99892
Copying blob sha256:b17cc31e431beb2f39988dff23d04f85ba4b446fc0a13f304774852fa3c87d85
Copying blob sha256:0fe19df8b8f8eb5f545f50e11f958bf37d27ab7da5c260a9bd2bc0ff6fb760a4
Copying blob sha256:0095ff73bb219f8cc528e9ddee70c2f84839ce2f55093e87e8048c4c9f201d2d
Copying blob sha256:d0a65b1da30968e03a4cb1ac8f2d76c6656b78dd4f0a79a3a539d4041e998e6f
Copying blob sha256:e4396ca14c332c42630f88b21aa9688e0b10b10dd4a3028ce875279632434b1b
Copying blob sha256:afcb32a7d9aa7e9b4bf7a270938d51c7e6b6658f0b53e59365e9397bebff7ce7
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
Copying config sha256:97c5724eaf16b1d36ae06e4b17d9230cc7b6078e74f72f3e7a7faa7640ce7aa3
Writing manifest to image destination
Copying config sha256:97c5724eaf16b1d36ae06e4b17d9230cc7b6078e74f72f3e7a7faa7640ce7aa3
Writing manifest to image destination
Storing signatures
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

This is great because with docker it would require a daemon and I wasn't able to run in Jenkins slaves in this Kubernetes cluster, but since podman doesn't require a daemon it fits perfect.

giuseppe commented 5 years ago

that is great to hear! :-)

rhatdan commented 5 years ago

@Noed We are working on getting buildah to do this without requiring --privilege. And of course I would prefer you to run buildah or podman inside of Podman or CRI-O.

Noed commented 5 years ago

@rhatdan that would be nice.

Turns out I don't have a chance to run my initial image in podman or buildah, because my workstation is Win10 :cry:

And on IBM cloud, seems the Kubernetes cluster uses containerd as the engine.

rhatdan commented 5 years ago

Hopefully that will change in the future. @Noed Install Fedora on the laptop and then you will never look back.