search

Praqma / memory-map-plugin

A repository for the memory-map-plugin
13 stars 16 forks source link

Docker doesn't mount volume when executing the command from a test #16

Closed andrius-ordojan closed 7 years ago

andrius-ordojan commented 7 years ago

The same script works if copied to a Jenkins pipeline job using the gui, but doesn't act in a reasonable manner when ran from test.

The job is created like so.

WorkflowJob job = jenkinsRule.jenkins.createProject(WorkflowJob.class, "p");
CpsFlowDefinition flowDef = new CpsFlowDefinition(s, false);
job.setDefinition(flowDef);

Here are the key parts from the script that describe the error

`sh 'ls -al && pwd'`

results in

+ ls -al
total 52
drwxrwxr-x 4 praqma praqma 4096 Nov 18 14:44 .
drwxrwxr-x 4 praqma praqma 4096 Nov 18 14:44 ..
-rwxrwxr-x 1 praqma praqma 2843 Nov 18 14:44 blink.c
-rw-rw-r-- 1 praqma praqma  629 Nov 18 14:44 Dockerfile
drwxrwxr-x 8 praqma praqma 4096 Nov 18 14:44 .git
drwxrwxr-x 2 praqma praqma 4096 Nov 18 14:44 include
-rwxrwxr-x 1 praqma praqma 2346 Nov 18 14:44 led.c
-rwxrwxr-x 1 praqma praqma  924 Nov 18 14:44 led.h
-rwxrwxr-x 1 praqma praqma 1312 Nov 18 14:44 makefile
-rw-rw-r-- 1 praqma praqma 4589 Nov 18 14:44 README.md
-rwxrwxr-x 1 praqma praqma  450 Nov 18 14:44 run.sh
-rwxrwxr-x 1 praqma praqma 1421 Nov 18 14:44 viperlite.ld
+ pwd
/tmp/hudson284166928905262158test/workspace/p
docker run -v `pwd`:/data $DKRIMG ls -al /data

evaluates to

docker run -v /tmp/hudson284166928905262158test/workspace/p:/data arm-none-eabi-gcc_4.8.4_hello_world:latest ls -al /data

results in

drwxr-xr-x  2 root root 4096 Nov  4 09:47 .
drwxr-xr-x 66 root root 4096 Nov 18 13:44 ..
wbagdon commented 7 years ago

I'd recommend adding docker pipeline plugin as a test dependency and take advantage of that pipeline syntax

https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/index.html#docker-workflow

wbagdon commented 7 years ago

Your POM dependencies should look like this.

<dependencies>
    <dependency>
        <groupId>net.praqma</groupId>
        <artifactId>praqmajutils</artifactId>
        <version>0.1.34</version>
    </dependency>
    <dependency>
        <groupId>org.antlr</groupId>
        <artifactId>antlr4</artifactId>
        <version>4.1</version>
    </dependency>
    <dependency>
        <groupId>com.stanfy</groupId>
        <artifactId>gson-xml-java</artifactId>
        <version>0.1.7</version>
    </dependency>
    <dependency>
        <groupId>org.jenkins-ci.plugins</groupId>
        <artifactId>git</artifactId>
        <version>3.0.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.jenkins-ci.plugins.workflow</groupId>
        <artifactId>workflow-aggregator</artifactId>
        <version>2.4</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.jenkins-ci.plugins</groupId>
        <artifactId>docker-workflow</artifactId>
        <version>1.9.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.jenkins-ci.plugins</groupId>
        <artifactId>structs</artifactId>
        <version>1.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>1.9.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.jenkins-ci.main</groupId>
        <artifactId>jenkins-test-harness</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.4</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.4.6</version>
        <scope>test</scope>
        <type>jar</type>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>2.4.6</version>
        <type>jar</type>
    </dependency>
    <dependency>
        <groupId>org.jenkins-ci.plugins</groupId>
        <artifactId>job-dsl</artifactId>
        <version>1.37</version>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.jenkins-ci</groupId>
        <artifactId>symbol-annotation</artifactId>
        <version>1.5</version>
    </dependency>
</dependencies>

Then your script can be:

node {
  git branch: 'arm-none-eabi-gcc_4.8.4_hello_world', url: 'https://github.com/Praqma/memory-map-examples'
  def myEnv = docker.build 'arm-none-eabi-gcc_4.8.4_hello_world:latest'
  myEnv.inside {
    sh 'ls -l'
  }
}
andrius-ordojan commented 7 years ago

Hi, thanks for the help. I've implemented your changes, but the result is still the same.

This is the script

node {
 git branch: 'arm-none-eabi-gcc_4.8.4_hello_world', url: 'https://github.com/Praqma/memory-map-examples'
    sh 'ls -al'
    sh 'pwd'
  def myEnv = docker.build 'arm-none-eabi-gcc_4.8.4_hello_world:latest'
  myEnv.inside {
    sh 'ls -al'
  }
}

The output. I picked out the highlights, but I'll post the full output bellow

[p] Running shell script
+ ls -al
total 52
drwxrwxr-x 4 praqma praqma 4096 Nov 24 13:50 .
drwxrwxr-x 4 praqma praqma 4096 Nov 24 13:50 ..
-rwxrwxr-x 1 praqma praqma 2843 Nov 24 13:50 blink.c
-rw-rw-r-- 1 praqma praqma  629 Nov 24 13:50 Dockerfile
drwxrwxr-x 8 praqma praqma 4096 Nov 24 13:50 .git
drwxrwxr-x 2 praqma praqma 4096 Nov 24 13:50 include
-rwxrwxr-x 1 praqma praqma 2346 Nov 24 13:50 led.c
-rwxrwxr-x 1 praqma praqma  924 Nov 24 13:50 led.h
-rwxrwxr-x 1 praqma praqma 1312 Nov 24 13:50 makefile
-rw-rw-r-- 1 praqma praqma 4589 Nov 24 13:50 README.md
-rwxrwxr-x 1 praqma praqma  450 Nov 24 13:50 run.sh
-rwxrwxr-x 1 praqma praqma 1421 Nov 24 13:50 viperlite.ld
[p] Running shell script
+ pwd
/tmp/hudson1036221874070707675test/workspace/p

[p] Running shell script
+ docker build -t arm-none-eabi-gcc_4.8.4_hello_world:latest .
Sending build context to Docker daemon 124.9 kB

Step 1 : FROM gcc:4.8.5
 ---> 4e41303e90cb
Step 2 : RUN wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2014-q3-update/+download/gcc-arm-none-eabi-4_8-2014q3-20140805-linux.tar.bz2   && tar -vjxf gcc-arm-none-eabi-4_8-2014q3-20140805-linux.tar.bz2 -C /opt    && rm -r gcc-arm-none-eabi-4_8-2014q3-20140805-linux.tar.bz2    && dpkg --add-architecture i386 && apt-get -y update && apt-get -y install ia32-libs
 ---> Using cache
 ---> 18f05a8af721
Step 3 : ENV PATH $PATH:/opt/gcc-arm-none-eabi-4_8-2014q3/bin"
 ---> Using cache
 ---> 96b983fdd13a
Step 4 : WORKDIR /data
 ---> Using cache
 ---> d2020a41c1e3
Successfully built d2020a41c1e3
[p] Running shell script
+ docker inspect -f . arm-none-eabi-gcc_4.8.4_hello_world:latest
.
$ docker run -t -d -u 1000:1000 -w /tmp/hudson1036221874070707675test/workspace/p -v /tmp/hudson1036221874070707675test/workspace/p:/tmp/hudson1036221874070707675test/workspace/p:rw -v /tmp/hudson1036221874070707675test/workspace/p@tmp:/tmp/hudson1036221874070707675test/workspace/p@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat arm-none-eabi-gcc_4.8.4_hello_world:latest
[p] Running shell script
+ ls -al
total 8
drwxr-xr-x 2 root root 4096 Nov 24 12:50 .
drwxr-xr-x 5 root root 4096 Nov 24 12:50 ..
$ docker stop --time=1 da9e0b2145ad0844c7a355b00bad730522a576fe5c9d97097f4e8991aea7861a
$ docker rm -f da9e0b2145ad0844c7a355b00bad730522a576fe5c9d97097f4e8991aea7861a

full output

Started
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] node
Running on master in /tmp/hudson1163508524827993872test/workspace/p
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] {
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] git
Cloning the remote Git repository
Cloning repository https://github.com/Praqma/memory-map-examples
 > git init /tmp/hudson1163508524827993872test/workspace/p # timeout=10
Fetching upstream changes from https://github.com/Praqma/memory-map-examples
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/Praqma/memory-map-examples +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/Praqma/memory-map-examples # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/Praqma/memory-map-examples # timeout=10
Fetching upstream changes from https://github.com/Praqma/memory-map-examples
 > git fetch --tags --progress https://github.com/Praqma/memory-map-examples +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/arm-none-eabi-gcc_4.8.4_hello_world^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/arm-none-eabi-gcc_4.8.4_hello_world^{commit} # timeout=10
Checking out Revision 68ad04e300ca6ac130c7cfef6c28a0f87d483496 (refs/remotes/origin/arm-none-eabi-gcc_4.8.4_hello_world)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 68ad04e300ca6ac130c7cfef6c28a0f87d483496
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b arm-none-eabi-gcc_4.8.4_hello_world 68ad04e300ca6ac130c7cfef6c28a0f87d483496
First time build. Skipping changelog.
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] sh
[p] Running shell script
+ ls -al
total 52
drwxrwxr-x 4 praqma praqma 4096 Nov 24 14:22 .
drwxrwxr-x 4 praqma praqma 4096 Nov 24 14:22 ..
-rwxrwxr-x 1 praqma praqma 2843 Nov 24 14:22 blink.c
-rw-rw-r-- 1 praqma praqma  629 Nov 24 14:22 Dockerfile
drwxrwxr-x 8 praqma praqma 4096 Nov 24 14:22 .git
drwxrwxr-x 2 praqma praqma 4096 Nov 24 14:22 include
-rwxrwxr-x 1 praqma praqma 2346 Nov 24 14:22 led.c
-rwxrwxr-x 1 praqma praqma  924 Nov 24 14:22 led.h
-rwxrwxr-x 1 praqma praqma 1312 Nov 24 14:22 makefile
-rw-rw-r-- 1 praqma praqma 4589 Nov 24 14:22 README.md
-rwxrwxr-x 1 praqma praqma  450 Nov 24 14:22 run.sh
-rwxrwxr-x 1 praqma praqma 1421 Nov 24 14:22 viperlite.ld
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] sh
[p] Running shell script
+ pwd
/tmp/hudson1163508524827993872test/workspace/p
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] sh
[p] Running shell script
+ docker build -t arm-none-eabi-gcc_4.8.4_hello_world:latest .
Sending build context to Docker daemon 124.9 kB

Step 1 : FROM gcc:4.8.5
 ---> 4e41303e90cb
Step 2 : RUN wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2014-q3-update/+download/gcc-arm-none-eabi-4_8-2014q3-20140805-linux.tar.bz2   && tar -vjxf gcc-arm-none-eabi-4_8-2014q3-20140805-linux.tar.bz2 -C /opt    && rm -r gcc-arm-none-eabi-4_8-2014q3-20140805-linux.tar.bz2    && dpkg --add-architecture i386 && apt-get -y update && apt-get -y install ia32-libs
 ---> Using cache
 ---> 18f05a8af721
Step 3 : ENV PATH $PATH:/opt/gcc-arm-none-eabi-4_8-2014q3/bin"
 ---> Using cache
 ---> 96b983fdd13a
Step 4 : WORKDIR /data
 ---> Using cache
 ---> d2020a41c1e3
Successfully built d2020a41c1e3
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] dockerFingerprintFrom
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] sh
[p] Running shell script
+ docker inspect -f . arm-none-eabi-gcc_4.8.4_hello_world:latest
.
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] withDockerContainer
$ docker run -t -d -u 1000:1000 -w /tmp/hudson1163508524827993872test/workspace/p -v /tmp/hudson1163508524827993872test/workspace/p:/tmp/hudson1163508524827993872test/workspace/p:rw -v /tmp/hudson1163508524827993872test/workspace/p@tmp:/tmp/hudson1163508524827993872test/workspace/p@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat arm-none-eabi-gcc_4.8.4_hello_world:latest
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] {
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] sh
[p] Running shell script
+ ls -al
total 8
drwxr-xr-x 2 root root 4096 Nov 24 13:22 .
drwxr-xr-x 5 root root 4096 Nov 24 13:22 ..
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] }
$ docker stop --time=1 8297435f39da0f55202a1d3bbb1f4376727ef3572abb6a20171281b4f123d338
$ docker rm -f 8297435f39da0f55202a1d3bbb1f4376727ef3572abb6a20171281b4f123d338
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] // withDockerContainer
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] }
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] // node
ha:AAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA[Pipeline] End of Pipeline
Finished: SUCCESS
wbagdon commented 7 years ago

Looks like it is mounting with -w /tmp/hudson1163508524827993872test/workspace/p

Can you cd /tmp/hudson1163508524827993872test/workspace/p inside the container? It should be stored in the WORKSPACE var or can be accessed with the Jenkins pwd pipeline command)

MadsNielsen commented 7 years ago

Right. This issue is quite confusing and it stems from a lack of understanding pipeline.

The issue, actually form the beginning has been a misconfigured pipelin script. The one below is working... What i fixed in it was:

  1. We don't want to clone the orginal memory map examples project.
  2. Instead we need to clone the repo where we've updated to the next commit. In this case it's a temp git repo on the test target.
  3. I've parameterized the deliverbranch and the deliverurl in the pipeline script. These are set in the test.
  4. I've removed the $BN env variable in the build script. It's now hardcoded, this is fine, since i suspect each compiler will have to be built and used differently.

node {
    git branch: '$deliverBranch', url: '$deliverUrl'
    sh '''
if [ "$(docker images | grep -c arm-none-eabi-gcc_4.8.4_hello_world)" -eq "0" ]; then
  docker build --rm -f Dockerfile -t arm-none-eabi-gcc_4.8.4_hello_world
fi
docker run -v `pwd`:/data arm-none-eabi-gcc_4.8.4_hello_world make clean all
'''
    memoryMap([gccParser(configurationFile: 'viperlite.ld', graphConfiguration: [[graphCaption: 'Memory sections', graphDataList: '.data,.bss,.text'], [graphCaption: 'Target memory', graphDataList: 'rom,ram']], mapFile: 'blink.map', parserTitle: 'GCC memory map', parserUniqueName: 'Gcc')])
}```

So this sum it up. Tests are passing on the fixUseCase branch and i'll get ready to release.
buep commented 7 years ago

Awesome Mads - before release, make sure everything is on master, branches are cleane, PR's merged etc. and let's then look at the release package together.

MadsNielsen commented 7 years ago

As an addendum...Jenkins fails VERY silently when the docker daemon is unresponsive. Another issue was that our docker installation was broken.

I've fixed our infrastructure on this as well....sigh!