Kaniko parallel builds

[tdensmore]

Actual behavior

I am trying to speed up the building of my containers using Jenkins and Kaniko. However Kaniko fails when I try to use parallel pipeline stages. Judging from the exception, it looks like Kaniko may temporarily lock the shared filesystem ?

[36mINFO[0m[0085] Taking snapshot of files...                  
error building image: error building stage: Failed to get file info for /app/examples/user-authentication-app/views/index.ejs: lstat /app/examples/user-authentication-app/views/index.ejs: no such file or directory

My serial pipeline using the same Kaniko stages works fine, but it takes about 15 minutes to run. I'm hoping to reduce this time using parallel stages, but so far it does not seem possible with Kaniko.

I'm hoping someone might be able to help, but I suspect this might be a current limitation of Kaniko...

Expected behavior

I expect that Kaniko would be able to build multiple Dockerfiles from the same repository in the same workspace.

To Reproduce

Run a parallel build stage using a simple Jenkinsfile.

[tejal29]

HI @tdensmore, are you running kaniko parallel builds on 2 separate pods? i am curious how you are using a shared filesystem .

We do use syscall.Sync() which might lock the shared filesystem.

[tdensmore]

I think Jenkins uses many containers in the build pod, and all share the workspace root. This Kaniko container gets the workspace mounted... and I assume parallel thread may compete for the filesystem resources?

Here is an example of my Jenkins declarative pipeline usage. I have not verified that this Jenkinsfile works, but I was using something similar.

Jenkinsfile.txt

[tejal29]

We do run kaniko pods in parallel in GKE. They run in different namespace and don't share the workspace root. And it run fines.

[joan-s-molas]

Same happens to us when running things in a step using parallel() in the same pod.

We have >20 images to build (microservices monorepo), and we run on EKS (limited pods due to VPC based CNI - unless you of course ditch the AWS CNI and switch to something else), so running parallel stages in a lot of pods would be a bit of an inconvenience.

[tdensmore]

A monolithic repo, especially when using golang is a common pattern.

@tejal29 are you using Jenkins, or a custom build tool? If Jenkins, can you explain how to modify my Jenkins pipeline to accommodate parallel using different pods / namespaces ?

[tejal29]

@tdensmore we are using different pods and not different containers in the same pod. I see the Jenkins pipeline parallel stages run containers in parallel in same pod. That is definitely not going to work right now since

• we use the same scratch workspace /kaniko
• parallel builds might end up writing in the same path. Deleting and adding files.

We can definitely make this experience better.

[hmaciastk]

For reference:

I also came across build problems when trying to build Dockerfile from within the same Kubernetes Jenkins slave using gcr.io/kaniko-project/executor:v1.5.0-debug image. The issue surfaced when using a parallel() construct in a Jenkinsfile and the same kaniko container defined by a containerTemplate(). I would get some builds to succeed and others to fail randomly. I got odd errors like this that didn't come up while using docker or sometimes while using Kaniko itself:

INFO[0013] cmd: /bin/sh INFO[0013] args: [-c addgroup testgroup && adduser -s /bin/bash -G testgroup -D testuser && apk add --no-cache bash gettext procps curl] INFO[0013] Running: [/bin/sh -c addgroup testgroup && adduser -s /bin/bash -G testgroup -D testuser && apk add --no-cache bash gettext procps curl] adduser: unknown group testgroup error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

The way I worked around this was to define multiple kaniko containers within my single Jenkins slave podTemplate() using containerTemplate() for each different build inside this Jenkins slave pod. Each parallel stage got to run in its own kaniko container. The odd error was no longer seen.

[gabyx]

I used something like

parallel {
  stage('Build build-container-go') {
    agent { kubernetes { yamlFile '.jenkins/pods/build.yaml' } }
    steps {
      container('build') {
          sh 'build-container-Y/build/run-kaniko.sh "$(pwd)" --no-push'
      }
    }
  }
  stage('Build build-container-cpp') {
    agent { kubernetes { yamlFile '.jenkins/pods/build.yaml' } }
    steps {
      container('build') {
        sh 'build-container-X/build/run-kaniko.sh "$(pwd)" --no-push'
      }
    }
  }
}