marcellodesales commented 6 years ago

Building docker image for runtime null
[Pipeline] sh
[job] Running shell script
+ docker build -t tests:null -f Dockerfile.tests .
/tmp/jenkinsTests.tmp/jenkins8996889882339540707test/workspace/job@tmp/durable-57fedee8/script.sh: 2: /tmp/jenkinsTests.tmp/jenkins8996889882339540707test/workspace/job@tmp/durable-57fedee8/script.sh: docker: not found
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Tests Execution)
...
...
...
ERROR: script returned exit code 127
Finished: FAILURE

ndeloof commented 6 years ago

Could you please provide the Jenkinsfile you're trying to run here ?

marcellodesales commented 6 years ago

@ndeloof Mainly using a Pipeline syntax... We have a Jenkins instance, with Blueocean, deployed in a Kubernetes cluster using docker-in-docker.

Stages using Docker

    stage('Build Test Image') {
      steps {
        script {
          currentBuild.displayName = "Test Image"
          currentBuild.description = "Building the docker image for running the test cases"
        }
        echo "Building docker image for tests from build stage ${env.GIT_COMMIT}"
        sh "docker build -t tests:${env.GIT_COMMIT} -f ${paas.build.docker.dockerfile.runtime} --target builder ."
      }
    }

    stage('Tests Execution') {
      parallel {
        stage('Execute Unit Tests') {
          steps {
            script {
              currentBuild.displayName = "Unit Tests"
              currentBuild.description = "Running the unit tests cases"
            }
            sh "docker run --name tests-${env.GIT_COMMIT} tests:${env.GIT_COMMIT}"
            sh "docker start tests-${env.GIT_COMMIT}"
            sh "docker cp tests-${env.GIT_COMMIT}:/opt/build/target ."

            // https://jenkins.io/doc/book/pipeline/jenkinsfile/#advanced-scripted-pipeline#using-multiple-agents
            stash includes: '**/target/*', name: 'build'
          }
        }
        stage('Execute Integration Tests') {
          when {
            expression { paas.idpsIsEnabled == true }
          }
          steps {
            script {
              currentBuild.displayName = "Integration Tests"
              currentBuild.description = "Running the Integration tests cases"
            }
            sh "docker run --rm tests:${env.GIT_COMMIT} mvn -s settings.xml -Dtest=\"*IT,*IntegrationTest\" -P jacoco test"
          }
        }
      }
    }

schnatterer commented 6 years ago

The error can also be reproduced with a minimal Jenkinsfile such as this:

node {
    stage('docker') {
        docker.image('hello-world').run()
    }
}

Could we somehow mount the docker socket and mount or install the client binary into jenkinsfile-runner? I did this for a shared library, maybe this could be helpful. The client binary can be downloaded from here: https://download.docker.com/linux/static/stable/x86_64/

ndeloof commented 6 years ago

@marcellodesales afaict your main issue is env.GIT_COMMIT is not set when running pipeline with jenkinsfile-runner - which makes sense imho : you might have uncommitted local changes. Having support for environment variables could help as a workaround

@schnatterer your issue seems unrelated to initial reported one. IIUC you miss docker CLI in your execution context. It you suggest docker-pipeline plugin to install it, please report this as a RFE for this plugin.

schnatterer commented 6 years ago

@ndeloof The initial issues also states docker: not found so I reckoned the missing docker executable would be the main problem. If we use docker directly via sh docker ... (original issue) or via the docker step provided by the docker pipeline plugin shouldn't matter.

It would be helpful if we could run docker steps in jenkinsfile-runner.

ndeloof commented 6 years ago

I can't see any reason the jenkinsfile-runner could not run docker steps, it just need docker command line available in the current node, i.e local machine running jenkinsfile-runner.

Based on your comment :

Could we somehow mount the docker socket and mount or install the client binary into jenkinsfile-runner?

it sounds like you use jenkinsfile-runner from a docker image, not running directly as a CLI tool. Doing so you indeed need to customize the docker image to have docker CLI installed and grant access to docker infrastructure.

marcellodesales commented 6 years ago

@ndeloof @schnatterer I actually got it to work following the suggestions from https://forums.docker.com/t/using-docker-in-a-dockerized-jenkins-container/322. I could get it to work to just run a container with the following volumes:

Mount the /var/run/docker.sock
Mount the docker command

Solution

I built the image locally and tagged as marcellodesales/jenkinsfile-runner
Running a container and mounting the volumes described

This is actually an acceptable solution for my case using Docker for Mac to support development of the Pipeline.

$ docker run -v $(pwd):/workspace  \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v $(which docker):$(which docker) marcellodesales/jenkinsfile-runner
Started
Running in Durability level: PERFORMANCE_OPTIMIZED
[
...
...
[Pipeline] [Validate Test] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build Test Image)
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] echo
Building docker image for tests from build stage null
[Pipeline] sh
[job] Running shell script
+ docker build -t tests:null -f Dockerfile --target builder .
Sending build context to Docker daemon  13.75MB
Step 1/9 : FROM maven:3.2.5-jdk-8 as builder
 ---> 95dd59c15f5d
Step 2/9 : MAINTAINER marcello.desales@gmail.com
 ---> Using cache
 ---> e4edaeb48381
Step 3/9 : ADD ./pom.xml /opt/build/pom.xml
 ---> Using cache
 ---> 2ca95e0b276e
Step 4/9 : ADD ./settings.xml /opt/build/settings.xml
 ---> Using cache
 ---> 1c52ecafca2b
Step 5/9 : WORKDIR  /opt/build/
 ---> Using cache
 ---> e2f5d1a6467f
Step 6/9 : RUN mvn -s settings.xml -B -e -C -T 1C org.apache.maven.plugins:maven-dependency-plugin:3.0.2:go-offline
 ---> Using cache
 ---> 9212f3bd2e5b
Step 7/9 : ADD ./src /opt/build/src
 ---> f7e2cc9f9aa6
Step 8/9 : RUN mvn -s settings.xml install -P embedded -Dmaven.test.skip=true -B -e -o -T 1C verify
 ---> Running in 52b96eb70b06
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 4
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building spring-cloud-config-server 1.1.6-SNAPSHOT
[INFO] ---------------------------------------------------
...
...
...
[INFO] --- maven-failsafe-plugin:2.20.1:integration-test (integration-test) @ spring-cloud-config-server ---
[INFO] Tests are skipped.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 27.438 s (Wall Clock)
[INFO] Finished at: 2018-04-23T21:02:49+00:00
[INFO] Final Memory: 46M/216M
[INFO] ------------------------------------------------------------------------
Removing intermediate container 52b96eb70b06
 ---> bbefce46d3bd
Step 9/9 : CMD mvn -s settings.xml -Dtest="!*IT,!*IntegrationTest" -P jacoco test
 ---> Running in aa7ab08da632
Removing intermediate container aa7ab08da632
 ---> a65a5a81ee21
Successfully built a65a5a81ee21
Successfully tagged tests:null
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Tests Execution)
[Pipeline] parallel
[Pipeline] [Execute Unit Tests] { (Branch: Execute Unit Tests)
[Pipeline] [Execute Integration Tests] { (Branch: Execute Integration Tests)
[Pipeline] [Execute Unit Tests] stage

jenkinsci / jenkinsfile-runner

Is it possible to actual run the docker steps? #17

Stages using Docker

Solution