search

microsoft / aroworkshop

Azure Red Hat OpenShift workshop
Creative Commons Attribution 4.0 International
58 stars 123 forks source link

/usr/libexec/s2i/run: line 485: exec: java: not found when deploying the CICD demo #40

Closed sjbylo closed 4 years ago

sjbylo commented 4 years ago

At the start of the CICO lab, I'm seeing the following after running:

./cicd/scripts/provision.sh deploy --enable-che --ephemeral --project-suffix aro
# oc get po -w 
NAME                        READY     STATUS              RESTARTS   AGE
che-5694c4c564-sg5kw        0/1       ContainerCreating   0          3m
cicd-demo-installer-44g4k   0/1       Error               0          5m
cicd-demo-installer-g6cnv   0/1       Error               0          4m
cicd-demo-installer-vhqj9   0/1       ContainerCreating   0          4s
gogs-1-cd2jx                1/1       Running             0          3m
gogs-postgresql-1-8gsbh     1/1       Running             0          3m
jenkins-2-692gx             0/1       Running             3          4m
jenkins-2-deploy            1/1       Running             0          4m
nexus-1-8lv27               1/1       Running             0          2m
sonardb-1-hfbnx             1/1       Running             0          2m
sonarqube-1-nqw8j           1/1       Running             0          2m

jenkins-2-692gx   0/1       Error     3         4m
cicd-demo-installer-vhqj9   1/1       Running   0         6s
jenkins-2-692gx   0/1       CrashLoopBackOff   3         4m
jenkins-2-692gx   0/1       Running   4         5m
jenkins-2-692gx   0/1       Error     4         5m
cicd-demo-installer-vhqj9   0/1       Error     0         55s
jenkins-2-692gx   0/1       CrashLoopBackOff   4         5m
odl_user@Azure:~$ oc logs jenkins-2-692gx
OpenShift client and server versions are oc v3.11.146
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.30.0.1:443
openshift v3.11.129
kubernetes v1.11.0+d4cacc0
alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License.

usage: alternatives --install <link> <name> <path> <priority>
                    [--initscript <service>]
                    [--family <family>]
                    [--slave <link> <name> <path>]*
       alternatives --remove <name> <path>
       alternatives --auto <name>
       alternatives --config <name>
       alternatives --display <name>
       alternatives --set <name> <path>
       alternatives --list

common options: --verbose --test --help --usage --version --keep-missing
                --altdir <directory> --admindir <directory>
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac
/usr/local/bin/jenkins-common.sh: line 33: java: command not found
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
Administrative monitors that contact the update center will remain active
Detected password environment variable change, updating Jenkins configuration ...
config.xml
Migrating slave image configuration to current version tag ...
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war
/usr/libexec/s2i/run: line 485: exec: java: not found
nichochen commented 4 years ago

@sjbylo I tried it out today with my ARO cluster, it works for me. Did you perform any other actions, or the error showed up right after deploying the demo?

$oc get pod -n cicd-aro
NAME                        READY     STATUS              RESTARTS   AGE
che-5ffcb75f4b-fmhtg        1/1       Running             0          11m
cicd-demo-installer-kfcfl   0/1       Completed           0          10m
cicd-demo-installer-xjn2n   0/1       Error               0          12m
gogs-1-rlg9v                1/1       Running             2          11m
gogs-postgresql-1-xmd65     1/1       Running             0          11m
jenkins-2-m7nns             1/1       Running             0          12m
maven-bkwj5                 1/1       Running             0          5m
nexus-2-hprgw               1/1       Running             0          11m
skopeo-3sp5w                0/1       ContainerCreating   0          29s
sonardb-1-ftfc9             1/1       Running             0          11m
sonarqube-1-b4f4z           1/1       Running             0          11m

Jenkins deployment config

$oc get dc jenkins -n cicd-aro -o yaml|grep -i stream -A 3
        kind: ImageStreamTag
        name: jenkins:2
        namespace: openshift
      lastTriggeredImage: docker-registry.default.svc:5000/openshift/jenkins@sha256:386f7a43af90a79eabe521be05fc6c2fad873c14b6dd53bbac38dc5418837f39

Jenkins image stream

$oc  describe is  -n openshift jenkins|tail

2 (latest)
  tagged from registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins

  * registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:386f7a43af90a79eabe521be05fc6c2fad873c14b6dd53bbac38dc5418837f39
      2 weeks ago

Jenkins pipeline build

$oc get build -n cicd-aro
NAME               TYPE              FROM      STATUS     STARTED          DURATION
tasks-pipeline-1   JenkinsPipeline             Complete   13 minutes ago
nichochen commented 4 years ago

I have reproduced the issue after recreating a new ARO cluster.

Jenkins image stream

$oc  describe is  -n openshift jenkins|tail

2 (latest)
  tagged from registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins

  * registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:da3488fb622ca393bcdb635e4d824f57708a7dc11db01ab3171c0d38ccb98474
      11 minutes ago
nichochen commented 4 years ago

A quick dirty work around is to change the jenkins deployment config, point the image to the old version of Jenkins.

$oc set triggers dc/jenkins --remove --from-image jenkins:2 -n cicd-aro
$oc set image dc/jenkins  jenkins=registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:386f7a43af90a79eabe521be05fc6c2fad873c14b6dd53bbac38dc5418837f39 -n cicd-aro

After the change you should see the dc is pointing to the new image

$oc get dc jenkins -o yam -n cicd-arol |grep image
        image: registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:386f7a43af90a79eabe521be05fc6c2fad873c14b6dd53bbac38dc5418837f39
sabbour commented 4 years ago

@nichochen do you know the root cause of this?

cloudmelon commented 4 years ago

Hi @nichochen I've point to the elder version of Jenkins, didn't work out : image

Logs :

image

Any thoughts ?

nichochen commented 4 years ago

@cloudmelon please verify the image in the dc is pointing to the old version.

oc describe dc jenkins -n cicd-aro

Before setting the image, we need to remove the image change trigger. I have updated the commands in this comment.

nichochen commented 4 years ago

@sabbour I will investigate this and update the workshop instructions accordingly.

cloudmelon commented 4 years ago

@nichochen The image change trigger is a great point, learn it ! Working now. thanks :) @sabbour I've been tried some old images ( https://access.redhat.com/containers/?tab=tags#/registry.access.redhat.com/openshift3/jenkins-2-rhel7 ) are working.

sabbour commented 4 years ago

Related bug https://bugzilla.redhat.com/show_bug.cgi?id=1754968

The working image is registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11.141. The bug report mentions this has been fixed in jenkins-2-rhel7:v3.11.152, but the release of that image is expected to rollout October 21st.

nichochen commented 4 years ago

I have verified this today, the issue have been fixed, Jenkins started without error.

cloudmelon commented 4 years ago

Thanks Nicolas :)

Best wishes from the UK Melony

On 10 Nov 2019, at 09:46, Nicholas Chen notifications@github.com wrote:

 I have verified this today, the issue have been fixed, Jenkins started without error.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

sjbylo commented 4 years ago

Excellent indeed! :)

On Sun, 10 Nov 2019, 18:16 Melony QIN, notifications@github.com wrote:

Thanks Nicolas :)

Best wishes from the UK Melony

On 10 Nov 2019, at 09:46, Nicholas Chen notifications@github.com wrote:

 I have verified this today, the issue have been fixed, Jenkins started without error.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/aroworkshop/issues/40?email_source=notifications&email_token=AAIYXY4OYC6GD42LOQD3JJTQS7NQPA5CNFSM4I3EM4GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDUZ4LQ#issuecomment-552181294, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIYXYZOV63ZIPW6VQJLDTDQS7NQPANCNFSM4I3EM4GA .