jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.58k stars 788 forks source link

Gradle-builder and Maven-builder pods do not find commands while being executed from Jenkins X #2256

Closed vjda closed 6 years ago

vjda commented 6 years ago

Summary

Hi all, first of all, I love Jenkins X. It's awesome, so thanks guys! When Jenkins X launchs a gradle-builder pod to execute a job which invokes any gradle command always throws an error such as:

[prueba-gradle] Running shell script
+ gradle--version
/home/jenkins/workspace/prueba-gradle@tmp/durable-fc0b96ff/script.sh: line 1: gradle: not found

However, if I connect to the pod using jx rsh gradle-id I can execute gradle commands as expected. I'm very confused because I'm using a fresh-installation with the default configuration.

Steps to reproduce the behavior

This is my pipeline which reproduces the issue:

pipeline {
  agent {
    label 'jenkins-gradle'
  }
  stages {
    stage('gradle') {
      steps {
          sh "env | sort"
          sh "gradle --version || echo '/opt/gradle not found'"
          sh "ls -la /opt/gradle || echo '/opt/gradle not found'"
          sh "whereis gradle || echo 'gradle not found'"
          sh "sleep 30000"
      }
    }
  }
}

Console output:

Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor

Agent gradle-sg8n1 is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (jenkins-gradle): 
* [gradle] jenkinsxio/builder-gradle:0.1.74(resourceRequestCpu: 400m, resourceRequestMemory: 512Mi, resourceLimitCpu: 1, resourceLimitMemory: 1024Mi)
* [jnlp] jenkinsci/jnlp-slave:3.26-1-alpine(resourceRequestCpu: 100m, resourceRequestMemory: 128Mi)

Running on gradle-sg8n1 in /home/jenkins/workspace/prueba-gradle
[Pipeline] {

[Pipeline] stage
[Pipeline] { (gradle)
[Pipeline] sh
[prueba-gradle] Running shell script
+ env
+ sort
AGENT_WORKDIR=/home/jenkins/agent
BUILD_DISPLAY_NAME=#7
BUILD_ID=7
BUILD_NUMBER=7
BUILD_TAG=jenkins-prueba-gradle-7
BUILD_URL=http://jenkins.jx.************.nip.io/job/prueba-gradle/7/
DOCKER_CONFIG=/home/jenkins/.docker/
DOCKER_REGISTRY=*************.dkr.ecr.eu-west-1.amazonaws.com
EXECUTOR_NUMBER=0
GIT_AUTHOR_EMAIL=jenkins-x@googlegroups.com
GIT_AUTHOR_NAME=jenkins-x-bot
GIT_COMMITTER_EMAIL=jenkins-x@googlegroups.com
GIT_COMMITTER_NAME=jenkins-x-bot
HEAPSTER_PORT=tcp://172.20.72.80:8082
HEAPSTER_PORT_8082_TCP=tcp://172.20.72.80:8082
HEAPSTER_PORT_8082_TCP_ADDR=172.20.72.80
HEAPSTER_PORT_8082_TCP_PORT=8082
HEAPSTER_PORT_8082_TCP_PROTO=tcp
HEAPSTER_SERVICE_HOST=172.20.72.80
HEAPSTER_SERVICE_PORT=8082
HOME=/home/jenkins
HOSTNAME=gradle-sg8n1
HUDSON_COOKIE=6aa30cc5-e402-4c33-b53d-ca816c1451e6
HUDSON_HOME=/var/jenkins_home
HUDSON_SERVER_COOKIE=646792a1e89de95c
HUDSON_URL=http://jenkins.jx.*************.nip.io/
JAVA_ALPINE_VERSION=8.171.11-r0
JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk
JAVA_VERSION=8u171
JENKINS_AGENT_NAME=gradle-sg8n1
JENKINS_AGENT_PORT=tcp://172.20.23.25:50000
JENKINS_AGENT_PORT_50000_TCP=tcp://172.20.23.25:50000
JENKINS_AGENT_PORT_50000_TCP_ADDR=172.20.23.25
JENKINS_AGENT_PORT_50000_TCP_PORT=50000
JENKINS_AGENT_PORT_50000_TCP_PROTO=tcp
JENKINS_AGENT_SERVICE_HOST=172.20.23.25
JENKINS_AGENT_SERVICE_PORT=50000
JENKINS_AGENT_SERVICE_PORT_SLAVELISTENER=50000
JENKINS_HOME=/var/jenkins_home
JENKINS_NAME=gradle-sg8n1
JENKINS_NODE_COOKIE=f8869ff3-a0d0-4e99-9f91-25ea206dbfb1
JENKINS_PORT=tcp://172.20.76.203:8080
JENKINS_PORT_8080_TCP=tcp://172.20.76.203:8080
JENKINS_PORT_8080_TCP_ADDR=172.20.76.203
JENKINS_PORT_8080_TCP_PORT=8080
JENKINS_PORT_8080_TCP_PROTO=tcp
JENKINS_SECRET=9ce34712184b9c649c225968f517442501c4ae7bec80cd8c75519f625f9c5d33
JENKINS_SERVER_COOKIE=durable-6df6e7abaa655bf49253a515b0d938c0
JENKINS_SERVICE_HOST=172.20.76.203
JENKINS_SERVICE_PORT=8080
JENKINS_SERVICE_PORT_HTTP=8080
JENKINS_TUNNEL=jenkins-agent:50000
JENKINS_URL=http://jenkins.jx.****************.nip.io/
JENKINS_X_CHARTMUSEUM_PORT=tcp://172.20.71.165:8080
JENKINS_X_CHARTMUSEUM_PORT_8080_TCP=tcp://172.20.71.165:8080
JENKINS_X_CHARTMUSEUM_PORT_8080_TCP_ADDR=172.20.71.165
JENKINS_X_CHARTMUSEUM_PORT_8080_TCP_PORT=8080
JENKINS_X_CHARTMUSEUM_PORT_8080_TCP_PROTO=tcp
JENKINS_X_CHARTMUSEUM_SERVICE_HOST=172.20.71.165
JENKINS_X_CHARTMUSEUM_SERVICE_PORT=8080
JENKINS_X_CHARTMUSEUM_SERVICE_PORT_HTTP=8080
JENKINS_X_DOCKER_REGISTRY_PORT=tcp://172.20.3.91:5000
JENKINS_X_DOCKER_REGISTRY_PORT_5000_TCP=tcp://172.20.3.91:5000
JENKINS_X_DOCKER_REGISTRY_PORT_5000_TCP_ADDR=172.20.3.91
JENKINS_X_DOCKER_REGISTRY_PORT_5000_TCP_PORT=5000
JENKINS_X_DOCKER_REGISTRY_PORT_5000_TCP_PROTO=tcp
JENKINS_X_DOCKER_REGISTRY_SERVICE_HOST=172.20.3.91
JENKINS_X_DOCKER_REGISTRY_SERVICE_PORT=5000
JENKINS_X_DOCKER_REGISTRY_SERVICE_PORT_REGISTRY=5000
JENKINS_X_MONGODB_PORT=tcp://172.20.71.126:27017
JENKINS_X_MONGODB_PORT_27017_TCP=tcp://172.20.71.126:27017
JENKINS_X_MONGODB_PORT_27017_TCP_ADDR=172.20.71.126
JENKINS_X_MONGODB_PORT_27017_TCP_PORT=27017
JENKINS_X_MONGODB_PORT_27017_TCP_PROTO=tcp
JENKINS_X_MONGODB_SERVICE_HOST=172.20.71.126
JENKINS_X_MONGODB_SERVICE_PORT=27017
JENKINS_X_MONGODB_SERVICE_PORT_MONGODB=27017
JENKINS_X_MONOCULAR_API_PORT=tcp://172.20.127.123:80
JENKINS_X_MONOCULAR_API_PORT_80_TCP=tcp://172.20.127.123:80
JENKINS_X_MONOCULAR_API_PORT_80_TCP_ADDR=172.20.127.123
JENKINS_X_MONOCULAR_API_PORT_80_TCP_PORT=80
JENKINS_X_MONOCULAR_API_PORT_80_TCP_PROTO=tcp
JENKINS_X_MONOCULAR_API_SERVICE_HOST=172.20.127.123
JENKINS_X_MONOCULAR_API_SERVICE_PORT=80
JENKINS_X_MONOCULAR_API_SERVICE_PORT_MONOCULAR_API=80
JENKINS_X_MONOCULAR_PRERENDER_PORT=tcp://172.20.42.157:80
JENKINS_X_MONOCULAR_PRERENDER_PORT_80_TCP=tcp://172.20.42.157:80
JENKINS_X_MONOCULAR_PRERENDER_PORT_80_TCP_ADDR=172.20.42.157
JENKINS_X_MONOCULAR_PRERENDER_PORT_80_TCP_PORT=80
JENKINS_X_MONOCULAR_PRERENDER_PORT_80_TCP_PROTO=tcp
JENKINS_X_MONOCULAR_PRERENDER_SERVICE_HOST=172.20.42.157
JENKINS_X_MONOCULAR_PRERENDER_SERVICE_PORT=80
JENKINS_X_MONOCULAR_PRERENDER_SERVICE_PORT_PRERENDER=80
JENKINS_X_MONOCULAR_UI_PORT=tcp://172.20.178.105:80
JENKINS_X_MONOCULAR_UI_PORT_80_TCP=tcp://172.20.178.105:80
JENKINS_X_MONOCULAR_UI_PORT_80_TCP_ADDR=172.20.178.105
JENKINS_X_MONOCULAR_UI_PORT_80_TCP_PORT=80
JENKINS_X_MONOCULAR_UI_PORT_80_TCP_PROTO=tcp
JENKINS_X_MONOCULAR_UI_SERVICE_HOST=172.20.178.105
JENKINS_X_MONOCULAR_UI_SERVICE_PORT=80
JENKINS_X_MONOCULAR_UI_SERVICE_PORT_MONOCULAR_UI=80
JOB_BASE_NAME=prueba-gradle
JOB_DISPLAY_URL=http://jenkins.jx.****************.nip.io/job/prueba-gradle/display/redirect
JOB_NAME=prueba-gradle
JOB_URL=http://jenkins.jx.****************.nip.io/job/prueba-gradle/
KUBERNETES_PORT=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=172.20.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=C.UTF-8
LD_LIBRARY_PATH=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64
NEXUS_PORT=tcp://172.20.218.222:80
NEXUS_PORT_80_TCP=tcp://172.20.218.222:80
NEXUS_PORT_80_TCP_ADDR=172.20.218.222
NEXUS_PORT_80_TCP_PORT=80
NEXUS_PORT_80_TCP_PROTO=tcp
NEXUS_SERVICE_HOST=172.20.218.222
NEXUS_SERVICE_PORT=80
NEXUS_SERVICE_PORT_NEXUS=80
NODE_LABELS=gradle-sg8n1 jenkins-gradle
NODE_NAME=gradle-sg8n1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
PWD=/home/jenkins/workspace/prueba-gradle
RUN_CHANGES_DISPLAY_URL=http://jenkins.jx.****************.nip.io/job/prueba-gradle/7/display/redirect?page=changes
RUN_DISPLAY_URL=http://jenkins.jx.****************.nip.io/job/prueba-gradle/7/display/redirect
SHLVL=3
STAGE_NAME=gradle
TILLER_NAMESPACE=kube-system
WORKSPACE=/home/jenkins/workspace/prueba-gradle
XDG_CONFIG_HOME=/home/jenkins
_JAVA_OPTIONS=-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx192m
[Pipeline] sh

[prueba-gradle] Running shell script
+ gradle --version
/home/jenkins/workspace/prueba-gradle@tmp/durable-6f01a060/script.sh: line 1: gradle: not found
+ echo '/opt/gradle not found'
/opt/gradle not found
[Pipeline] sh
[prueba-gradle] Running shell script
+ ls -la /opt/gradle
ls: /opt/gradle: No such file or directory
+ echo '/opt/gradle not found'
/opt/gradle not found
[Pipeline] sh
[prueba-gradle] Running shell script
+ whereis gradle
/home/jenkins/workspace/prueba-gradle@tmp/durable-56b698f6/script.sh: line 1: whereis: not found
+ echo 'gradle not found'
gradle not found
[Pipeline] sh

[prueba-gradle] Running shell script
+ sleep 30000

As you may notice, Jenkins X cannot find the path /opt/gradle where gradle should be located. Besides, when I connect directly to the pod, the PATH enviroment var is different than the displayed in the Jenkins console. I mean:

⋊> jx rsh                                                                                                                                              19:35:06? Pick Pod: gradle-6htx2
Defaulting container name to gradle.
Use 'kubectl describe pod/gradle-6htx2 -n jx' to see all of the containers in this pod.
bash-4.2# env | grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/google-cloud-sdk/bin:/opt/google/chrome:/opt/android-sdk-linux/tools:/opt/android-sdk-linux/tools/bin:/opt/android-sdk-linux/platform-tools
bash-4.2# ls /opt/gradle/
bin  getting-started.html  init.d  lib  LICENSE  media  NOTICE
bash-4.2# gradle --version
Picked up _JAVA_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx192m
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

------------------------------------------------------------
Gradle 4.6
------------------------------------------------------------

Build time:   2018-02-28 13:36:36 UTC
Revision:     8fa6ce7945b640e6168488e4417f9bb96e4ab46c

Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_191 (Oracle Corporation 25.191-b12)
OS:           Linux 4.14.72-73.55.amzn2.x86_64 amd64

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.565
jenkins x platform 0.0.2902
Kubernetes cluster v1.10.3-eks
kubectl            v1.12.1
helm client        v2.10.0+g9ad53aa
helm server        v2.10.0+g9ad53aa
git                git version 2.17.1

Kubernetes cluster

What kind of Kubernetes cluster are you using & how did you create it? I'm using an EKS cluster and it was created with terraform and then JenkinsX installed with jx install --provider=eks

Operating system / Environment

EKS with Kubernetes version v1.10.3

Expected behavior

Pod builders should be able to execute gradle or maven commands.

Actual behavior

Gradle and Maven pods do not find gradle and maven binaries.

Thanks in advance!!

vjda commented 6 years ago

After digging into the issue I realized I didn't select the container in the pipeline and if you don't specify it, Kubernetes plugin assumes as default container the jnlp. That's why Jenkins X could not find the gradle and maven binaries... 😓

Do you know if the default container can be selected in The Kubernetes Pod Template section? If the pod template is defined in the pipeline this can be done using the flag defaultContainer. See an example:

pipeline {
  agent {
    kubernetes {
      label 'mypod'
      customWorkspace 'some/other/path'
      defaultContainer 'maven'
      yamlFile 'KubernetesPod.yaml'
    }
  }

  stages {
    stage('Run maven') {
      steps {
        sh 'mvn -version'
        sh "echo Workspace dir is ${pwd()}"
      }
    }
  }
}

Reference https://github.com/jenkinsci/kubernetes-plugin#declarative-pipeline

vjda commented 6 years ago

In order to this might be helpul for others, a default container can be configured in a pipeline as follow:

pipeline {
  agent {
    kubernetes {
      label 'jenkins-gradle'
      defaultContainer 'gradle'
    }
  }
  stages {
    stage('gradle') {
      steps {
          sh "gradle --version"
      }
    }
  }
}

I can close the issue. Cheers!