Create a Jenkins pipeline plugin

[jromero]

Description

As a Jenkins user, I would like to use a Cloud Native Buildpacks builder in my pipeline to generate a Docker image.

Proposed solution

Create a pipeline plugin that makes it easy for users to use Cloud Native Buildpacks within Jenkins.

As an example, see the Docker pipeline plugin.

Describe alternatives you've considered

Standard Jenkins Plugin

An alternative to a pipeline plugin, a standard plugin can be provided in which the user fills out a few input boxes. This functionality could potentially be added to the same plugin if possible.

As an example, see Docker build/publish plugin.

Sample configuration

Without implementing anything, we could leverage existing plugins to compose together a working sample on how Cloud Native buildpacks would be integrated.

Additional context

• About Pipelines
• Reference Docker pipeline plugin
• Reference Docker build/publish plugin

Open Questions

• Can a single plugin provide both standard and pipeline DSL functionality?

[jromero]

@2pai has volunteered to work on this as part of GSoC.

[abarke]

Any updates here? Would be great to use this plugin. Is there any other way to currently use pack CLI and jenkins to run tests within a built docker container?

[jromero]

Unfortunately the GSoC project did not go through.

@2pai is this something you'd still be interested in pursuing even outside of GSoC? I would be open to providing more hands on assistance if necessary.

[jromero]

@abarke we don't have any examples but theoretically it should be possible. As long as a container runtime is available you may use one of the following options.

1. If you are able to configure how containers are executed and share volumes between tasks: You should be able to use the lifecycle similar to how Tekton uses it.
   • We recently added a single executable (creator) that does all the phases in one. That might be easier but don't have a working example of that yet and documentation is sparse at the moment.
2. If the task (within the host) has access to the daemon socket: You should be able to use pack CLI just like you would locally.

[2pai]

Unfortunately the GSoC project did not go through.

@2pai is this something you'd still be interested in pursuing even outside of GSoC? I would be open to providing more hands on assistance if necessary.

Sure, but I think I need some assistance. is that okay ?

[jromero]

@2pai,

Yes, of course. Feel free to schedule some time using this link or reach me directly via Slack.

[abarke]

@jromero I want to try option 2. as you mentioned above by using pack on our build agents.

How would that play together with the rest of jenkins... could I just write a script that builds the docker container using pack and then fire it up using docker run and finally docker exec to ssh into the container in order to run my test suite? Not sure what the best approach is here. Any rough ideas?

[dfreilich]

@abarke At this point, we actually publish a docker image with pack (https://hub.docker.com/r/buildpacksio/pack), which should at least help with this.

[dfreilich]

@jromero Thinking about this, this probably should be either in the buildpacks/ci or buildpacks/community repo. What do you think makes most sense?

[abarke]

@abarke At this point, we actually publish a docker image with pack (https://hub.docker.com/r/buildpacksio/pack), which should at least help with this.

sounds promising... do you have a workflow documentation to implement such a solution in jenkins?

I would love to see this plugin get off the ground, it would really help to push pack and buildpacks.io adoption in our company and would definitely solve a lot of PHP environment issues we are having with Jenkins. We are using pack for creating docker images for local development and it works a treat! Then we deploy the same app to Heroku. The missing piece is the CI integration. If we can use pack to setup our environment in a docker container in jenkins, that would be the icing on the cake and finally we will have a decent setup for future apps.

[dfreilich]

@abarke Thanks for the feedback! I don't know much about Jenkins plugins, but this seems to say that you should be able to do something like:

pipeline {
    agent {
        docker { image 'buildpacksio/pack:latest' }
    }
    stages {
        stage('Test') {
            steps {
                sh 'pack build ...'
            }
        }
    }
}

to use pack in CI. Let me know if that helps!

Also, the Buildpacks project is undergoing review to move from CNCF sandbox to CNCF incubation, and they're specifically looking to hear about companies/users who are end-users of the CNB project. If you'd be able to reply to this thread here, and also perhaps comment here (https://github.com/buildpacks/community/issues/12), that would be a tremendous help towards the project growth!

cc @jkutner @hone @sclevine

[abarke]

@dfreilich I logged into that forum using the first link but there is no way I can see to reply to that thread. I will however post a message in the adopters github chat ;)

[abarke]

Are there any efforts to create a jenkins plugin currently?

[jromero]

@abarke thanks for the follow up. I just updated the labels. At this point in time, I don't believe anyone is going through the effort. I'm more than happy to guide/mentor if anyone is interested in tackling this integration.

[pandarrr]

In case someone is having trouble, this is my setup to run pack in Jenkins.

pipeline {
  agent {
    docker {
      image '<preferred_distro>'
      args '--name ${BUILD_TAG}'
    }
  }
  stages {
    stage('Package') {
      steps {
        sh '''docker run \
                --user root \
                --volumes-from "${BUILD_TAG}" \
                buildpacksio/pack:latest build "<image_name>"
                  --path "${WORKSPACE}"'''
      }
    }
  }
}

Our Jenkins runs in a container and each pipeline has a separate container(s). The docker sock is mounted automatically in our configuration. BUILD_TAG and WORKSPACE are provided by Jenkins.

Naming the root agent, passing the volumes along with --volumes-from, and setting --path were all necessary to give pack access to the actual workspace. Running as --user root was necessary for pack to access docker.sock.

Using the pack image as an agent didn't work because Jenkins expects the entrypoint to be a shell. There may be more setup if you use --publish but we push the image in a separate step.

[jjsheridan]

@pandarrr Can you provide some details about your setup? I have docker.sock and /usr/bin/docker exposed to Jenkins with volume mounts and can run Docker in the Jenkins terminal, but I get "docker: not found" when I hit sh '''docker run step.