nsenter: failed to sync with parent: SYNC_USERMAP_ACK: got 255: Invalid argument

[Amos-85]

Issue:

When trying to build image with img on kubernetes from Jenkins pipeline,
there are exceptions below in error section.

Details:

kubernetes version: 1.19.8
img version: v0.5.11, build 5b90868
Jenkins: 2.293
OS: Ubuntu 20.04.2 LTS

unprivileged_userns_clone: enabled in OS

Error:

newuidmap: write to uid_map failed: Invalid argument
nsenter: failed to use newuidmap: Invalid argument
nsenter: failed to sync with parent: SYNC_USERMAP_ACK: got 255: Invalid argument

pod yaml:

apiVersion: "v1"
kind: "Pod"
metadata:
  name: jenkins-agent
  annotations:
    container.apparmor.security.beta.kubernetes.io/img: unconfined
    container.seccomp.security.alpha.kubernetes.io/img: unconfined
spec:
  activeDeadlineSeconds: 108000
  containers:
  - name: img
    securityContext:
      runAsUser: 1000
    image: r.j3ss.co/img
    args:
    - "cat"
    command:
    - "/bin/sh"
    - "-c"
    imagePullPolicy: "IfNotPresent"
    resources:
      limits:
        memory: "2Gi"
        cpu: "700m"
      requests:
        memory: "2Gi"
        cpu: "600m"
    tty: true
    workingDir: "/home/jenkins/agent"

Jenkins pipeline:

pipeline {
    agent {
        kubernetes {
            inheritFrom 'jenkins-agent'
        }
    }
    options {
        skipStagesAfterUnstable()
    }

    stages {
        stage('Build') {
            steps {
                container('img'){        
                    sh "strace -Z img build . --tag some-tag"
                }
            }
        }
    }
}

When trying to exec ( kubectl -n jenkins-builds exec -it jenkins-agent -c img -- sh ) the img container in the pod and run img build, img build is running successfully with those warnings:

WARN[0000] Process sandbox is not available, consider unmasking procfs: mount: permission denied (are you root?) 
WARN[0000] using host network as the default

Running img in Jenkins with strace -Z:

+ strace -Z img build . --tag some-tag
newfstatat(AT_FDCWD, "/usr/local/sbin/unpigz", 0xc0000b9218, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/bin/unpigz", 0xc0000b92e8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/sbin/unpigz", 0xc0000b93b8, 0) = -1 ENOENT (No such file or directory)
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=792451072, u64=140480582768640}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc0001dd9e4) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=792451072, u64=140480582768640}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc0001ddb7c) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=792451072, u64=140480582768640}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc0001dd8a4) = -1 EPERM (Operation not permitted)
newfstatat(AT_FDCWD, "/etc/mdns.allow", 0xc000322d38, 0) = -1 ENOENT (No such file or directory)
futex(0x1d8c868, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x1d8c868, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=792451072, u64=140480582768640}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc00050925c) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=792450656, u64=140480582768224}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 7, 0xc00050921c) = -1 EPERM (Operation not permitted)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=73, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=792451072, u64=140480582768640}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc000509544) = -1 EPERM (Operation not permitted)
futex(0x1d8c868, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x1d8c868, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
newuidmap: write to uid_map failed: Invalid argument
nsenter: failed to use newuidmap: Invalid argument
nsenter: failed to sync with parent: SYNC_USERMAP_ACK: got 255: Invalid argument
+++ exited with 24 +++

What may cause this issue specifically when running img from Jenkins ?

[Amos-85]

There was a USER override Envar which cause this issue.