nodejs extension: Failed to extend build image

cmoulliard commented 1 year ago

Issue

When the tekton Buildspack extension pipelineRun is executed on RHTAP, then a nodejs build is raising this error.


error: Error -1 running transaction
Timer: Extender ran for 17.308097547s and ended at 2023-08-28T07:34:03Z
[31;1mERROR: [0mfailed to extend build image: extending build image: applying Dockerfile to image: error building stage: failed to execute command: waiting for process to exit: exit status 1
...
Full log : https://gist.github.com/cmoulliard/12e8db5dcf3fd6f1a8c5cf88a50e4b1a#file-gistfile1-txt-L346-L348 (edited)

cmoulliard commented 1 year ago

From Ozzy

google suggests that error might be from (micro) dnf failing to complete the install.. I do spot a couple of errors re groupadd not being known further up, maybe the image needs shadow-utils ..

@mhdawson Does the current ubi paketo stack install shadow-utils ? doesn't look like it.. although .. if this IS the error, it kinda feels like whatever rpm being installed that's failing to find groupadd, has a missing dependency on shadow-utils.

FYI: @BarDweller

cmoulliard commented 1 year ago

Does the current ubi paketo stack install shadow-utils ? Correct me if I say something wrong but I dont think that ubi paketo installs shadow-utils. See what it is currently installed : https://github.com/paketo-community/ubi-nodejs-extension/blob/367b5451a2917f001cb3e56139691606ad52cc86/generate.go#L19

As the Tekton pipeline using nodejs extension is working locally on kind+tekton on my macbook, why does it fail when it is executed on RHTAP + ocp4 ? Is it because kind bootstrap a k8s cluster on ubuntu 2023-08-23 16:50:05 Welcome to Debian GNU/Linux 11 (bullseye)! vs RHEL for ocp4 ?

Remark: I executed locally the tekton pipeline and extension phase installs well the following package:

[buildpacks : extender] Installing: shadow-utils;2:4.6-17.el8;x86_64;ubi-8-baseos-rpms

I dont see such error messages locally

Installing: libsemanage;2.9-9.el8_6;x86_64;ubi-8-baseos-rpms
Installing: shadow-utils;2:4.6-17.el8;x86_64;ubi-8-baseos-rpms
/var/tmp/rpm-tmp.h9eb9g: line 2: /usr/sbin/groupadd: No such file or directory
/var/tmp/rpm-tmp.h9eb9g: line 3: /usr/sbin/groupadd: No such file or directory
Installing: libutempter;1.1.6-14.el8;x86_64;ubi-8-baseos-rpms
...

but instead

[buildpacks : extender] Installing: libsemanage;2.9-9.el8_6;x86_64;ubi-8-baseos-rpms
[buildpacks : extender] Installing: shadow-utils;2:4.6-17.el8;x86_64;ubi-8-baseos-rpms
[buildpacks : extender] Installing: libutempter;1.1.6-14.el8;x86_64;ubi-8-baseos-rpms

mhdawson commented 1 year ago

the list of what is installed is here: https://github.com/paketo-community/ubi-nodejs-extension/blob/926ce866b8142996dda3cadaefa5c2233c3df852/generate.go#L19. It is not specifically installed as it is not in that list.

It may be a depdenency of one of the packages that is installed.

Is the list of "Installing: XXX" exactly the same in the two cases up until you see the failure?

cmoulliard commented 1 year ago

Is the list of "Installing: XXX" exactly the same in the two cases up until you see the failure?

From a quick comparison, the lists are the same from test executed on RHTAP vs locally.

cmoulliard commented 1 year ago

I will try to run the test case on ocp4 + tekton to see what we have as error and if this is related to runAsUser: 0 and runAsGroup: 0

mhdawson commented 1 year ago

After setting up an environment were I could reproduce and discussion swith @cmoulliard, this is my understanding of where this one stands:

It is likely that that the problem is that the task that runs the extension step needs to run as root and the pipeline is not set up to do that quite right. @cmoulliard came to that same conclusion through his investigation today as well.
even just dnf installing shadow-utils and nothing else fails with an error. I think this could be reproduced without the buildpacks in a task that just tries to dnf isntall shadow-utils.
It’s git that pulls in shadow-utils. Without that the pipeline builds/publishes the image for the application ok.

Next step is that @cmoulliard needs to get some help from Ops Container engineers to figure out how to set up the environment properly.

cmoulliard commented 1 year ago

The problem could be easily reproduced without using Buildpack as mentioned by @mhdawson. Use the following PipelineRun and deploy it on a minikube OR k8s kind cluster vs ocp4 and you will see that process is working on local k8s cluster

[install-tools : install-shadow-utils] + microdnf --setopt=install_weak_deps=0 --setopt=tsflags=nodocs install -y shadow-utils
[install-tools : install-shadow-utils]
[install-tools : install-shadow-utils] (microdnf:23): librhsm-WARNING **: 11:30:39.082: Found 0 entitlement certificates
[install-tools : install-shadow-utils]
[install-tools : install-shadow-utils] (microdnf:23): librhsm-WARNING **: 11:30:39.101: Found 0 entitlement certificates
[install-tools : install-shadow-utils] Downloading metadata...
[install-tools : install-shadow-utils] Downloading metadata...
[install-tools : install-shadow-utils] Downloading metadata...
[install-tools : install-shadow-utils] Package                                           Repository            Size
[install-tools : install-shadow-utils] Installing:
[install-tools : install-shadow-utils]  audit-libs-3.0.7-4.el8.x86_64                    ubi-8-baseos-rpms 125.9 kB
[install-tools : install-shadow-utils]  basesystem-11-5.el8.noarch                       ubi-8-baseos-rpms  10.8 kB
[install-tools : install-shadow-utils]  bash-4.4.20-4.el8_6.x86_64                       ubi-8-baseos-rpms   1.6 MB
[install-tools : install-shadow-utils]  bzip2-libs-1.0.6-26.el8.x86_64                   ubi-8-baseos-rpms  49.1 kB
[install-tools : install-shadow-utils]  ca-certificates-2022.2.54-80.2.el8_6.noarch      ubi-8-baseos-rpms 942.8 kB
[install-tools : install-shadow-utils]  chkconfig-1.19.1-1.el8.x86_64                    ubi-8-baseos-rpms 203.1 kB
[install-tools : install-shadow-utils]  coreutils-8.30-15.el8.x86_64                     ubi-8-baseos-rpms   1.3 MB
[install-tools : install-shadow-utils]  coreutils-common-8.30-15.el8.x86_64              ubi-8-baseos-rpms   2.1 MB
...
[install-tools : install-shadow-utils] Installing: coreutils;8.30-15.el8;x86_64;ubi-8-baseos-rpms
[install-tools : install-shadow-utils] Installing: ca-certificates;2022.2.54-80.2.el8_6;noarch;ubi-8-baseos-rpms
[install-tools : install-shadow-utils] Installing: shadow-utils;2:4.6-17.el8;x86_64;ubi-8-baseos-rpms
[install-tools : install-shadow-utils] Complete.

but fails on ocp4 cluster

...
[install-tools : install-shadow-utils] + microdnf --setopt=install_weak_deps=0 --setopt=tsflags=nodocs install -y shadow-utils
[install-tools : install-shadow-utils]
[install-tools : install-shadow-utils] (microdnf:17): libdnf-CRITICAL **: 11:27:22.862: History database cannot be created, using in-memory database instead: SQLite error on "/var/lib/dnf/history.sqlite": Open failed: unable to open database file
[install-tools : install-shadow-utils] Downloading metadata...
[install-tools : install-shadow-utils] Downloading metadata...
[install-tools : install-shadow-utils] Downloading metadata...
[install-tools : install-shadow-utils] Package                                           Repository            Size
[install-tools : install-shadow-utils] Installing:
...
[install-tools : install-shadow-utils] Installing: ca-certificates;2022.2.54-80.2.el8_6;noarch;ubi-8-baseos-rpms
[install-tools : install-shadow-utils] Installing: shadow-utils;2:4.6-17.el8;x86_64;ubi-8-baseos-rpms
[install-tools : install-shadow-utils] error: Error -1 running transaction

Script used:

cat <<'EOF' | kubectl create -f -
---
apiVersion: v1
imagePullSecrets:
- name: dockercfg
kind: ServiceAccount
metadata:
  name: sa-with-secrets
secrets:
- name: dockercfg
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: shadow-utils
spec:
  serviceAccountName: sa-with-secrets
  pipelineSpec:
    tasks:
      - name: install-tools
        taskSpec:
          volumes:
            - name: task-volume
              emptyDir: {}

          steps:
            - name: install-shadow-utils
              image: paketocommunity/builder-ubi-base
              script: |
                #!/usr/bin/env bash
                set -eux

                microdnf  --setopt=install_weak_deps=0 --setopt=tsflags=nodocs install -y shadow-utils

              securityContext:
                runAsUser: 0
                runAsGroup: 0

              volumeMounts:
                - name: task-volume
                  mountPath: /var/
---
EOF

tkn pr logs shadow-utils -f

cmoulliard commented 1 year ago

We can even reproduce the problem without using Tekton.

kubectl delete deployment/dummy-container

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy
  labels:
    app: dummy
spec:
  selector:
    matchLabels:
      app: dummy
  replicas: 1
  template:
    metadata:
      labels:
        app: dummy
    spec:
      containers:
      - name: install-shadow-utils
        image: registry.access.redhat.com/ubi8/ubi-minimal:8.8
        command:
          - sh
        args:
          - '-c'
          - >
           microdnf install -y shadow-utils

        securityContext:
          runAsUser: 0
          runAsGroup: 0

        volumeMounts:
          - name: var-vol
            mountPath: /var/
          - name: usr-vol
            mountPath: /usr/share/info
      serviceAccountName: "sa-with-secrets"
      volumes:
       - name: var-vol
         emptyDir: {}
       - name: usr-vol
         emptyDir: {}
EOF

kubectl rollout status deployment/dummy
kubectl logs -lapp=dummy -f

cmoulliard commented 1 year ago

There is something that I dont understand on ocp4. We are getting the error even if: uid=0(root) gid=0(root) groups=0(root)

Log of the execution of the pod

./scripts/tekton/pod-shadow-utils
deployment.apps "dummy" deleted
deployment.apps/dummy created
Waiting for deployment "dummy" rollout to finish: 0 of 1 updated replicas are available...
deployment "dummy" successfully rolled out
Linux dummy-86d784c7f9-fjpbd 5.14.0-284.18.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
uid=0(root) gid=0(root) groups=0(root)

(microdnf:1): librhsm-WARNING **: 13:02:01.361: Found 0 entitlement certificates

(microdnf:1): librhsm-WARNING **: 13:02:01.362: Found 0 entitlement certificates
Downloading metadata...
Downloading metadata...
Downloading metadata...
Package                                                   Repository            Size
Installing:
 audit-libs-3.0.7-4.el8.x86_64                            ubi-8-baseos-rpms 125.9 kB
 basesystem-11-5.el8.noarch                               ubi-8-baseos-rpms  10.8 kB
 bash-4.4.20-4.el8_6.x86_64                               ubi-8-baseos-rpms   1.6 MB
 bzip2-libs-1.0.6-26.el8.x86_64                           ubi-8-baseos-rpms  49.1 kB
 ca-certificates-2022.2.54-80.2.el8_6.noarch              ubi-8-baseos-rpms 942.8 kB
 chkconfig-1.19.1-1.el8.x86_64                            ubi-8-baseos-rpms 203.1 kB
 coreutils-8.30-15.el8.x86_64                             ubi-8-baseos-rpms   1.3 MB
 coreutils-common-8.30-15.el8.x86_64                      ubi-8-baseos-rpms   2.1 MB
 crypto-policies-20221215-1.gitece0092.el8.noarch         ubi-8-baseos-rpms  65.6 kB
 crypto-policies-scripts-20221215-1.gitece0092.el8.noarch ubi-8-baseos-rpms  85.9 kB
 expat-2.2.5-11.el8.x86_64                                ubi-8-baseos-rpms 116.1 kB
 filesystem-3.8-6.el8.x86_64                              ubi-8-baseos-rpms   1.1 MB
 gawk-4.2.1-4.el8.x86_64                                  ubi-8-baseos-rpms   1.2 MB
 gdbm-1:1.18-2.el8.x86_64                                 ubi-8-baseos-rpms 132.8 kB
 gdbm-libs-1:1.18-2.el8.x86_64                            ubi-8-baseos-rpms  61.8 kB
 glibc-2.28-225.el8.x86_64                                ubi-8-baseos-rpms   2.3 MB
 glibc-all-langpacks-2.28-225.el8.x86_64                  ubi-8-baseos-rpms  26.8 MB
 glibc-common-2.28-225.el8.x86_64                         ubi-8-baseos-rpms   1.0 MB
 glibc-gconv-extra-2.28-225.el8.x86_64                    ubi-8-baseos-rpms   1.6 MB
 gmp-1:6.1.2-10.el8.x86_64                                ubi-8-baseos-rpms 329.2 kB
 grep-3.1-6.el8.x86_64                                    ubi-8-baseos-rpms 280.4 kB
 info-6.5-7.el8.x86_64                                    ubi-8-baseos-rpms 203.3 kB
 keyutils-libs-1.5.10-9.el8.x86_64                        ubi-8-baseos-rpms  34.8 kB
 krb5-libs-1.18.2-25.el8_8.x86_64                         ubi-8-baseos-rpms 862.6 kB
 libacl-2.2.53-1.el8.x86_64                               ubi-8-baseos-rpms  35.6 kB
 libattr-2.4.48-3.el8.x86_64                              ubi-8-baseos-rpms  27.6 kB
 libcap-2.48-5.el8_8.x86_64                               ubi-8-baseos-rpms  76.1 kB
 libcap-ng-0.7.11-1.el8.x86_64                            ubi-8-baseos-rpms  34.2 kB
 libcom_err-1.45.6-5.el8.x86_64                           ubi-8-baseos-rpms  50.6 kB
 libffi-3.1-24.el8.x86_64                                 ubi-8-baseos-rpms  38.6 kB
 libgcc-8.5.0-18.el8.x86_64                               ubi-8-baseos-rpms  83.1 kB
 libnsl2-1.2.0-2.20180605git4a062cf.el8.x86_64            ubi-8-baseos-rpms  59.1 kB
 libselinux-2.9-8.el8.x86_64                              ubi-8-baseos-rpms 169.7 kB
 libsemanage-2.9-9.el8_6.x86_64                           ubi-8-baseos-rpms 172.1 kB
 libsepol-2.9-3.el8.x86_64                                ubi-8-baseos-rpms 348.1 kB
 libsigsegv-2.11-5.el8.x86_64                             ubi-8-baseos-rpms  31.0 kB
 libtasn1-4.13-4.el8_7.x86_64                             ubi-8-baseos-rpms  78.0 kB
 libtirpc-1.1.4-8.el8.x86_64                              ubi-8-baseos-rpms 115.8 kB
 libverto-0.3.2-2.el8.x86_64                              ubi-8-baseos-rpms  24.6 kB
 libxcrypt-4.1.1-6.el8.x86_64                             ubi-8-baseos-rpms  74.5 kB
 mpfr-3.1.6-1.el8.x86_64                                  ubi-8-baseos-rpms 226.7 kB
 ncurses-6.1-9.20180224.el8.x86_64                        ubi-8-baseos-rpms 396.4 kB
 ncurses-base-6.1-9.20180224.el8.noarch                   ubi-8-baseos-rpms  83.1 kB
 ncurses-libs-6.1-9.20180224.el8.x86_64                   ubi-8-baseos-rpms 341.8 kB
 openssl-1:1.1.1k-9.el8_7.x86_64                          ubi-8-baseos-rpms 726.7 kB
 openssl-libs-1:1.1.1k-9.el8_7.x86_64                     ubi-8-baseos-rpms   1.5 MB
 openssl-pkcs11-0.4.10-3.el8.x86_64                       ubi-8-baseos-rpms  67.7 kB
 p11-kit-0.23.22-1.el8.x86_64                             ubi-8-baseos-rpms 332.1 kB
 p11-kit-trust-0.23.22-1.el8.x86_64                       ubi-8-baseos-rpms 140.3 kB
 pcre-8.42-6.el8.x86_64                                   ubi-8-baseos-rpms 215.7 kB
 pcre2-10.32-3.el8_6.x86_64                               ubi-8-baseos-rpms 252.6 kB
 platform-python-3.6.8-51.el8_8.1.x86_64                  ubi-8-baseos-rpms  88.6 kB
 platform-python-pip-9.0.3-22.el8.noarch                  ubi-8-baseos-rpms   1.7 MB
 platform-python-setuptools-39.2.0-7.el8.noarch           ubi-8-baseos-rpms 647.4 kB
 popt-1.18-1.el8.x86_64                                   ubi-8-baseos-rpms  62.9 kB
 python3-libs-3.6.8-51.el8_8.1.x86_64                     ubi-8-baseos-rpms   8.2 MB
 python3-pip-wheel-9.0.3-22.el8.noarch                    ubi-8-baseos-rpms 916.3 kB
 python3-setuptools-wheel-39.2.0-7.el8.noarch             ubi-8-baseos-rpms 296.0 kB
 readline-7.0-10.el8.x86_64                               ubi-8-baseos-rpms 204.2 kB
 redhat-release-8.8-0.8.el8.x86_64                        ubi-8-baseos-rpms  46.3 kB
 sed-4.5-5.el8.x86_64                                     ubi-8-baseos-rpms 305.4 kB
 setup-2.12.2-9.el8.noarch                                ubi-8-baseos-rpms 185.3 kB
 shadow-utils-2:4.6-17.el8.x86_64                         ubi-8-baseos-rpms   1.3 MB
 sqlite-libs-3.26.0-18.el8_8.x86_64                       ubi-8-baseos-rpms 595.0 kB
 tzdata-2023c-1.el8.noarch                                ubi-8-baseos-rpms 488.2 kB
 xz-libs-5.2.4-4.el8_6.x86_64                             ubi-8-baseos-rpms  96.1 kB
 zlib-1.2.11-21.el8_7.x86_64                              ubi-8-baseos-rpms 105.4 kB
Transaction Summary:
 Installing:       67 packages
 Reinstalling:      0 packages
 Upgrading:         0 packages
 Obsoleting:        0 packages
 Removing:          0 packages
 Downgrading:       0 packages
Downloading packages...
Running transaction test...
Installing: python3-setuptools-wheel;39.2.0-7.el8;noarch;ubi-8-baseos-rpms
Installing: python3-pip-wheel;9.0.3-22.el8;noarch;ubi-8-baseos-rpms
Installing: tzdata;2023c-1.el8;noarch;ubi-8-baseos-rpms
Installing: redhat-release;8.8-0.8.el8;x86_64;ubi-8-baseos-rpms
Installing: setup;2.12.2-9.el8;noarch;ubi-8-baseos-rpms
Installing: filesystem;3.8-6.el8;x86_64;ubi-8-baseos-rpms
Installing: basesystem;11-5.el8;noarch;ubi-8-baseos-rpms
Installing: ncurses-base;6.1-9.20180224.el8;noarch;ubi-8-baseos-rpms
Installing: pcre2;10.32-3.el8_6;x86_64;ubi-8-baseos-rpms
Installing: libselinux;2.9-8.el8;x86_64;ubi-8-baseos-rpms
Installing: ncurses-libs;6.1-9.20180224.el8;x86_64;ubi-8-baseos-rpms
Installing: glibc-all-langpacks;2.28-225.el8;x86_64;ubi-8-baseos-rpms
Installing: glibc-gconv-extra;2.28-225.el8;x86_64;ubi-8-baseos-rpms
Installing: glibc-common;2.28-225.el8;x86_64;ubi-8-baseos-rpms
Installing: glibc;2.28-225.el8;x86_64;ubi-8-baseos-rpms
Installing: bash;4.4.20-4.el8_6;x86_64;ubi-8-baseos-rpms
Installing: libsepol;2.9-3.el8;x86_64;ubi-8-baseos-rpms
Installing: zlib;1.2.11-21.el8_7;x86_64;ubi-8-baseos-rpms
Installing: info;6.5-7.el8;x86_64;ubi-8-baseos-rpms
Installing: readline;7.0-10.el8;x86_64;ubi-8-baseos-rpms
Installing: gmp;1:6.1.2-10.el8;x86_64;ubi-8-baseos-rpms
Installing: libattr;2.4.48-3.el8;x86_64;ubi-8-baseos-rpms
Installing: libacl;2.2.53-1.el8;x86_64;ubi-8-baseos-rpms
Installing: sed;4.5-5.el8;x86_64;ubi-8-baseos-rpms
Installing: bzip2-libs;1.0.6-26.el8;x86_64;ubi-8-baseos-rpms
Installing: gdbm-libs;1:1.18-2.el8;x86_64;ubi-8-baseos-rpms
Installing: libcom_err;1.45.6-5.el8;x86_64;ubi-8-baseos-rpms
Installing: libffi;3.1-24.el8;x86_64;ubi-8-baseos-rpms
Installing: p11-kit;0.23.22-1.el8;x86_64;ubi-8-baseos-rpms
Installing: libxcrypt;4.1.1-6.el8;x86_64;ubi-8-baseos-rpms
Installing: gdbm;1:1.18-2.el8;x86_64;ubi-8-baseos-rpms
Installing: mpfr;3.1.6-1.el8;x86_64;ubi-8-baseos-rpms
Installing: coreutils-common;8.30-15.el8;x86_64;ubi-8-baseos-rpms
Installing: sqlite-libs;3.26.0-18.el8_8;x86_64;ubi-8-baseos-rpms
Installing: expat;2.2.5-11.el8;x86_64;ubi-8-baseos-rpms
Installing: keyutils-libs;1.5.10-9.el8;x86_64;ubi-8-baseos-rpms
Installing: libcap-ng;0.7.11-1.el8;x86_64;ubi-8-baseos-rpms
Installing: audit-libs;3.0.7-4.el8;x86_64;ubi-8-baseos-rpms
Installing: libsemanage;2.9-9.el8_6;x86_64;ubi-8-baseos-rpms
Installing: libsigsegv;2.11-5.el8;x86_64;ubi-8-baseos-rpms
Installing: gawk;4.2.1-4.el8;x86_64;ubi-8-baseos-rpms
Installing: libtasn1;4.13-4.el8_7;x86_64;ubi-8-baseos-rpms
Installing: libverto;0.3.2-2.el8;x86_64;ubi-8-baseos-rpms
Installing: ncurses;6.1-9.20180224.el8;x86_64;ubi-8-baseos-rpms
Installing: pcre;8.42-6.el8;x86_64;ubi-8-baseos-rpms
Installing: grep;3.1-6.el8;x86_64;ubi-8-baseos-rpms
Installing: popt;1.18-1.el8;x86_64;ubi-8-baseos-rpms
Installing: chkconfig;1.19.1-1.el8;x86_64;ubi-8-baseos-rpms
Installing: p11-kit-trust;0.23.22-1.el8;x86_64;ubi-8-baseos-rpms
Installing: xz-libs;5.2.4-4.el8_6;x86_64;ubi-8-baseos-rpms
Installing: libgcc;8.5.0-18.el8;x86_64;ubi-8-baseos-rpms
Installing: libcap;2.48-5.el8_8;x86_64;ubi-8-baseos-rpms
Installing: krb5-libs;1.18.2-25.el8_8;x86_64;ubi-8-baseos-rpms
Installing: libtirpc;1.1.4-8.el8;x86_64;ubi-8-baseos-rpms
Installing: libnsl2;1.2.0-2.20180605git4a062cf.el8;x86_64;ubi-8-baseos-rpms
Installing: platform-python-pip;9.0.3-22.el8;noarch;ubi-8-baseos-rpms
Installing: platform-python-setuptools;39.2.0-7.el8;noarch;ubi-8-baseos-rpms
Installing: python3-libs;3.6.8-51.el8_8.1;x86_64;ubi-8-baseos-rpms
Installing: platform-python;3.6.8-51.el8_8.1;x86_64;ubi-8-baseos-rpms
Installing: openssl;1:1.1.1k-9.el8_7;x86_64;ubi-8-baseos-rpms
Installing: crypto-policies-scripts;20221215-1.gitece0092.el8;noarch;ubi-8-baseos-rpms
Installing: crypto-policies;20221215-1.gitece0092.el8;noarch;ubi-8-baseos-rpms
Installing: openssl-pkcs11;0.4.10-3.el8;x86_64;ubi-8-baseos-rpms
Installing: openssl-libs;1:1.1.1k-9.el8_7;x86_64;ubi-8-baseos-rpms
Installing: coreutils;8.30-15.el8;x86_64;ubi-8-baseos-rpms
Installing: ca-certificates;2022.2.54-80.2.el8_6;noarch;ubi-8-baseos-rpms
Installing: shadow-utils;2:4.6-17.el8;x86_64;ubi-8-baseos-rpms
error: Error -1 running transaction

cmoulliard commented 1 year ago

I found an interesting error message if we install the same packages using dnf and not microdnf -->

error: unpacking of archive failed on file /usr/bin/newgidmap;64f6043e: cpio: cap_set_file failed - Inappropriate ioctl for device
error: shadow-utils-2:4.6-17.el8.x86_64: install failed

cmoulliard commented 1 year ago

FYI: I created 2 tickets; one to request that microdnf better logs the errors and the second to ask to the shadow-utils team why we got the cpio: cap_set_file failed error

@mhdawson @BarDweller

cmoulliard commented 1 year ago

Why do we have to install the package shadow-utils ? Is it needed as prereq to install nodejs, npm ? @mhdawson

BarDweller commented 1 year ago

at least for the prototype build/run images, we required shadow-utils to be able to have adduser and addgroup to create the user & group to use for the cnb_user_id and cnb_group_id .. we did experiment later with setting those to the values in the existing images.. I don't think we kept those changes.. it was always simpler to have 1000/1000 or 1001/1001 etc

BarDweller commented 1 year ago

as for cpio cap_set_file failed.. this has to be environmental based on that we don't see the error in other environments (eg, docker is ok, your local kind is ok etc).. googling for the error brings back stuff from 5+ years ago with fedora, but the error is explained as basically the filesystem doesnt support the operation being requested.. given we're now in 2023, I suspect this would come down to one of two possibilities.. 1) whatever storage type or filesystem is being used in the container env doesn't support those operations (feels unlikely, but plausible as I doubt this kind of operation is common).. or 2) the pod needs additional permissions granted to allow this type of operation..

did a quick google on "cpio: cap_set_file pod permissions" and came across https://github.com/containers/podman/issues/5364 which suggests a particular permission to grant.. but I suspect if we go that way we're going to play whack-a-mole with each perm.. and then https://discuss.linuxcontainers.org/t/cpio-cap-set-file/472 which suggests using a privileged container (which wouldn't check this stuff anyway). .. Privileged might be a bit 'too open' for the tastes of ocp/rhtap tho, so it may be worth trying the individual perm.

building apps in a pod where the pod is expected to install rpms is always going to need decent perms.. so I wonder where the middle ground sits between an app hosting cluster that has to be restricted for safety, and a dev cluster/build cluster that needs a little more freedom

cmoulliard commented 1 year ago

I created a scc to add more capabilities but without success

kubectl delete deployment/dummy
kubectl delete scc/my-custom-scc
kubectl delete sa/my-serviceaccount

cat <<'EOF' | kubectl apply -f -
---
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: my-custom-scc
allowPrivilegedContainer: false
defaultAddCapabilities: []
requiredDropCapabilities:
  - KILL
allowedCapabilities:
  - SETGID
  - SETUID
  - SETFCAP
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  annotations:
    serviceaccounts.openshift.io/scc: my-custom-scc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy
  labels:
    app: dummy
spec:
  selector:
    matchLabels:
      app: dummy
  replicas: 1
  template:
    metadata:
      labels:
        app: dummy
    spec:
      serviceAccountName: my-serviceaccount
      containers:
      - name: install-shadow-utils
        image: registry.access.redhat.com/ubi8/ubi
        command:
          - sh
        args:
          - '-c'
          - >
           id;
           dnf install -y shadow-utils

        securityContext:
          runAsUser: 0
          runAsGroup: 0

        volumeMounts:
          - name: var-vol
            mountPath: /var/
          - name: usr-vol
            mountPath: /usr/share/info
          - name: var-lib
            mountPath: /var/lib/containers

      volumes:
       - name: var-vol
         emptyDir: {}
       - name: usr-vol
         emptyDir: {}
       - name: var-lib
         emptyDir: {}
EOF

kubectl rollout status deployment/dummy
kubectl logs -lapp=dummy --follow=true

adambkaplan commented 11 months ago

Does buildpacks execute (micro-)dnf install within the main container, or a "sub-container" in the style of buildah?

Looking at our buildah task in the OpenShift Pipelines catalog, it only asks for SETFCAP capability. But this task assumes it is running with the pipeline service account, which can use an SCC that is similar to anyuid: https://github.com/tektoncd/operator/blob/e0507dc1f00d5a6a3e8a6d571e595ada0235ae90/cmd/openshift/operator/kodata/openshift/00-prereconcile/openshift-pipelines-scc.yaml

cmoulliard commented 11 months ago

Does buildpacks execute (micro-)dnf install within the main container, or a "sub-container" in the style of buildah?

Looking at our buildah task in the OpenShift Pipelines catalog, it only asks for SETFCAP capability. But this task assumes it is running with the pipeline service account, which can use an SCC that is similar to anyuid: https://github.com/tektoncd/operator/blob/e0507dc1f00d5a6a3e8a6d571e595ada0235ae90/cmd/openshift/operator/kodata/openshift/00-prereconcile/openshift-pipelines-scc.yaml

I think that it runs within the main container. Do you confirm @BarDweller ?

BarDweller commented 11 months ago

I'd suspect it's within the main container... we're talking about the extender lifecycle binary that internally uses Kaniko to extend the container that's running using Dockerfiles .. so unless Kaniko is doing something odd, its likely in the main container. I think you worked on the prototype code for this @cmoulliard ?

cmoulliard commented 11 months ago

I'd suspect it's within the main container... we're talking about the extender lifecycle binary that internally uses Kaniko to extend the container that's running using Dockerfiles .. so unless Kaniko is doing something odd, its likely in the main container. I think you worked on the prototype code for this @cmoulliard ?

I do my test using latest released lifecycle including extension

cmoulliard commented 11 months ago

I did a new test @adambkaplan using the scc definition you proposed and that fails too on ocp4. Here is the script file that I'm using to reproduce the error on ocp4 - https://console-openshift-console.apps.snowdrop.lab.upshift.rdu2.redhat.com/ -> https://github.com/redhat-buildpacks/testing/blob/main/scripts/shadow-utils-scc

kubectl delete deployment/dummy
kubectl delete sa/my-serviceaccount
kubectl delete scc/my-custom-scc

cat <<'EOF' | kubectl apply -f -
---
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: my-custom-scc
# allowPrivilegedContainer: false
# defaultAddCapabilities: []
# requiredDropCapabilities:
#   - KILL
# allowedCapabilities:
#   - SETGID
#   - SETUID
#   - SETFCAP
# runAsUser:
#   type: RunAsAny
# seLinuxContext:
#   type: MustRunAs
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities:
  - SETFCAP
defaultAddCapabilities: null
fsGroup:
  type: MustRunAs
groups:
  - system:cluster-admins
priority: 10
readOnlyRootFilesystem: false
requiredDropCapabilities:
  - MKNOD
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  annotations:
    serviceaccounts.openshift.io/scc: my-custom-scc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy
  labels:
    app: dummy
spec:
  selector:
    matchLabels:
      app: dummy
  replicas: 1
  template:
    metadata:
      labels:
        app: dummy
    spec:
      serviceAccountName: my-serviceaccount
      containers:
        - name: install-shadow-utils
          image: registry.access.redhat.com/ubi8/ubi
          command:
            - sh
          args:
            - '-c'
            - >
              id;
              dnf install -y shadow-utils

          securityContext:
            runAsUser: 0
            runAsGroup: 0

          volumeMounts:
            - name: var-vol
              mountPath: /var/
            - name: usr-vol
              mountPath: /usr/share/info
            - name: var-lib
              mountPath: /var/lib/containers

      volumes:
        - name: var-vol
          emptyDir: {}
        - name: usr-vol
          emptyDir: {}
        - name: var-lib
          emptyDir: {}
EOF

kubectl rollout status deployment/dummy
kubectl logs -lapp=dummy --follow=true

cmoulliard commented 11 months ago

I finally fixed the issue using the following config, part of the pod container.

REMARK: Passing a scc to a pod using the ServiceAccount is not the way to go according to : https://issues.redhat.com/browse/OCPBUGS-19439

oc new-project buildpacks
oc delete pod/dummy-1

cat <<EOF | oc apply -f -
---
apiVersion: v1
kind: Pod
metadata:
  name: dummy-1
spec:
  #serviceAccountName: my-custom-sa
  containers:
    - name: install-shadow-utils
      image: registry.access.redhat.com/ubi8/ubi-minimal
      command:
        - sh
      args:
        - '-c'
        - >
          microdnf install -y shadow-utils

      securityContext:
        runAsUser: 0
        runAsGroup: 0
        capabilities:
          add:
            - "SYS_ADMIN"
            - "SETFCAP"

      volumeMounts:
        - name: var-vol
          mountPath: /var/

  volumes:
    - name: var-vol
      emptyDir: {}
EOF

sleep 10
oc logs dummy-1 -f

Question: Should we use this securityContext's config for the extender running on k8s/ocp or explore another approach (which one) ? @adambkaplan @BarDweller @mhdawson

As we can expect, that will fail on RHTAP as not supported

Screenshot 2023-09-21 at 08 54 07

cgwalters commented 11 months ago

I only glanced at this but I think actually you don't want to grant CAP_SETFCAP to the container today, because it will just confuse things because without also configuring a user namespace, it may try to write capabilities greater than the bound.

(I may be wrong in some details of this...ultimately basically user namespaces are the real solution to a lot of things like this though)

cmoulliard commented 11 months ago

(I may be wrong in some details of this...ultimately basically user namespaces are the real solution to a lot of things like this though)

Can you elaborate what a user namespace is and how that could be created/managed on OpenShift, RHTAP ? @cgwalters

redhat-buildpacks / testing

nodejs extension: Failed to extend build image #42

Issue