Build of the apps/quarkus-sample-app fails

[cmoulliard]

Issue

The build of the apps/quarkus-sample-app project fails using quarkus-jvm stack

===> DETECTING
io.quarkus.buildpacks.buildpack 0.0.1
===> ANALYZING
Previous image with name "quarkus-app" not found
===> RESTORING
===> BUILDING
---> Quarkus Buildpack
Building /workspace :: Native? 0
Maven transfer progress disabled.
Building using mvn package -B --no-transfer-progress -DskipTests
/usr/bin/mvn: line 98: which: command not found
The JAVA_HOME environment variable is not defined correctly
This environment variable is needed to run this program
NB: JAVA_HOME should point to a JDK not a JRE
ERROR: failed to build: exit status 1
ERROR: failed to build: executing lifecycle: failed with status code: 51

Command executed within the project

pack build quarkus-app --path apps/quarkus-sample-app --builder redhat/buildpacks-builder-quarkus-jvm:latest

Additional information

[snowdrop@h01-121 quarkus-buildpacks]$ docker run --rm -it redhat/buildpacks-builder-quarkus-jvm:latest bash
[jboss@bf397dc271d3 layers]$ ls -la /usr/bin
total 17000
dr-xr-xr-x 1 root  root    4096 Aug 10 14:56  .
drwxrwxr-x 1 jboss root    4096 Aug 10 14:51  ..
-r-xr-xr-x 1 root  root      48 Apr 14  2020 '['
-rwxr-xr-x 1 root  root      29 Apr 21 14:07  alias
lrwxrwxrwx 1 root  root      26 Aug 10 14:53  alt-java -> /etc/alternatives/alt-java
-r-xr-xr-x 1 root  root      51 Apr 14  2020  arch
-rwxr-xr-x 1 root  root   33640 Feb  1  2021  aserver
lrwxrwxrwx 1 root  root       4 Dec  3  2020  awk -> gawk
-r-xr-xr-x 1 root  root      52 Apr 14  2020  b2sum
-r-xr-xr-x 1 root  root      53 Apr 14  2020  base32
-r-xr-xr-x 1 root  root      53 Apr 14  2020  base64
-r-xr-xr-x 1 root  root      55 Apr 14  2020  basename
-rwxr-xr-x 1 root  root 1150584 Apr 21 14:07  bash
lrwxrwxrwx 1 root  root      10 Apr 21 14:07  bashbug -> bashbug-64
...
-rwxr-xr-x 1 root  root      28 Apr 21 14:07  wait
-rwxr-xr-x 1 root  root   16936 May  4  2020  watchgnupg
-r-xr-xr-x 1 root  root      49 Apr 14  2020  wc
-r-xr-xr-x 1 root  root      50 Apr 14  2020  who
-r-xr-xr-x 1 root  root      53 Apr 14  2020  whoami
-rwxr-xr-x 1 root  root     246 Aug 24  2018  withsctp
-rwxr-xr-x 1 root  root     424 Aug 12  2018  x86_64-redhat-linux-gnu-pkg-config
-rwxr-xr-x 1 root  root   90320 Nov  5  2018  xargs
...

[jboss@bf397dc271d3 layers]$ ls -la /etc/alternatives/mvn
lrwxrwxrwx 1 root root 24 Aug 10 14:55 /etc/alternatives/mvn -> /usr/share/maven/bin/mvn

[jboss@bf397dc271d3 layers]$ ls -la /usr/share/maven/bin/mvn
-rwxr-xr-x 1 root root 5846 Dec 24  2020 /usr/share/maven/bin/mvn

[cmoulliard]

Workaround is to tag the ubi6 openjdk11 image to use version 1.3.

[quintesse]

The weird thing is that version 1.3 also doesn't have which installed! So why does that work? Is the mvn command different? Is the mvn command that is included in version 1.10 somehow incompatible? Questions questions.

Btw, instead of using version 1.3 I tried installing which and that also fixes the problem.

So I'm not really sure what the best solution is.

[cmoulliard]

I have investigated and here is what I think the problem is. The maven version packaged with 1.3 and 1.10 is the same and includes maven 3.6.2

docker run --rm -it registry.access.redhat.com/ubi8/openjdk-11:1.3 bash
[jboss@b869ddc1b9be ~]$ mvn -v
Apache Maven 3.6.2 (Red Hat 3.6.2-6)
Maven home: /usr/share/maven
Java version: 11.0.12, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.42.2.el7.x86_64", arch: "amd64", family: "unix"

The script which is execuded when we do a maven build will launch /usr/share/maven/bin/mvn where a test is taking place if JAVA_HOME is set or not

if [ -z "$JAVA_HOME" ] ; then
  JAVACMD=`which java`
else
  JAVACMD="$JAVA_HOME/bin/java"
fi

If JAVA_HOME is not set, then which java is executed, otherwise JAVACMD var is set to $JAVA_HOME/bin/java.

So, my first temporary conclusion is that the JAVA_HOME env var was not set using the latest ubi8-openjdk11 image (1.10 now) vs was set using ubi8-openjdk11:1-3.

Remark: I will do a new build using buildpack to very if JAVA_HOME warning are logged or not

FYI: @quintesse

[quintesse]

Hmm that's also weird because in both versions JAVA_HOME seems to be set correctly...

[cmoulliard]

in both versions JAVA_HOME seems to be set correctly

Wait wait. To verify that hypothesis, a new buildpack build is needed as the error occurs during buildpacks maven execution

[quintesse]

Indeed. It seems the pack process circumvents something in the normal startup because if I run env as part of the build command only ~5 ENV vars are defined, and JAVA_HOME is definintely not one of them!

[cmoulliard]

The problem is not related to JAVA_HOME (except if echo $JAVA_HOME executed by the buildpack bin/build do not work) as, in both cases, JAVA_HOME ENV VAR is empty -->

## 1.10

===> BUILDING
---> Quarkus Buildpack
Building /workspace :: Native? 0
Maven transfer progress disabled.
JAVA_HOME:
MAVEN_HOME:
Printenv
HOSTNAME=7f5f9ec1c191
CNB_STACK_ID=io.quarkus.buildpacks.stack.jvm
PWD=/workspace
HOME=/home/jboss
CNB_BUILDPACK_DIR=/cnb/buildpacks/io.quarkus.buildpacks.buildpack/0.0.1
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/s2i
_=/usr/bin/printenv
Building using mvn package -B --no-transfer-progress -DskipTests
/usr/bin/mvn: line 98: which: command not found
The JAVA_HOME environment variable is not defined correctly
This environment variable is needed to run this program
NB: JAVA_HOME should point to a JDK not a JRE
ERROR: failed to build: exit status 1
ERROR: failed to build: executing lifecycle: failed with status code: 51

# 1.3

===> BUILDING
---> Quarkus Buildpack
Building /workspace :: Native? 0
Maven transfer progress disabled.
JAVA_HOME:
MAVEN_HOME:
Printenv
HOSTNAME=9097fe6efe2f
CNB_STACK_ID=io.quarkus.buildpacks.stack.jvm
PWD=/workspace
HOME=/home/jboss
CNB_BUILDPACK_DIR=/cnb/buildpacks/io.quarkus.buildpacks.buildpack/0.0.1
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/s2i
_=/usr/bin/printenv
Building using mvn package -B --no-transfer-progress -DskipTests
[INFO] Scanning for projects...

[cmoulliard]

Another temporary workaround to avoid to install which is to set the JAVA_HOME within the bin/build file before to execute the mvnCommand

# Set JAVA_HOME
export JAVA_HOME="/usr/lib/jvm/java-11"
echo "JAVA_HOME set to $JAVA_HOME"

OR

# Set JAVA_HOME to avoid the error reported since 1.10 by maven
# /usr/bin/mvn: line 98: which: command not found
# The JAVA_HOME environment variable is not defined correctly
# This environment variable is needed to run this program
# NB: JAVA_HOME should point to a JDK not a JRE
export JAVA_HOME="/usr/lib/jvm/java-openjdk"
echo "JAVA_HOME set to $JAVA_HOME"

Ideally we should be able to retrieve it to avoid to hard code the PATH

ls -la /usr/lib/jvm/java-11
lrwxrwxrwx 1 root root 29 Jul 23 17:37 /usr/lib/jvm/java-11 -> /etc/alternatives/java_sdk_11
[jboss@a422a9ad1462 ~]$ ls -la /etc/alternatives/java_sdk_11
lrwxrwxrwx 1 root root 55 Jul 23 17:37 /etc/alternatives/java_sdk_11 -> /usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64

WDYT @quintesse

[quintesse]

Indeed, the best option would be to retrieve it somehow.

Which is actually why I wanted to find the script that sets the JAVA_HOME when you simply run docker run --rm -it redhat/buildpacks-builder-quarkus-jvm:latest env :

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/s2i
HOSTNAME=fa3381cd4cb8
TERM=xterm
container=oci
HOME=/home/jboss
JAVA_HOME=/usr/lib/jvm/java-11
JAVA_VENDOR=openjdk
JAVA_VERSION=11
JBOSS_CONTAINER_OPENJDK_JDK_MODULE=/opt/jboss/container/openjdk/jdk
AB_PROMETHEUS_JMX_EXPORTER_CONFIG=/opt/jboss/container/prometheus/etc/jmx-exporter-config.yaml
JBOSS_CONTAINER_PROMETHEUS_MODULE=/opt/jboss/container/prometheus
AB_JOLOKIA_AUTH_OPENSHIFT=true
AB_JOLOKIA_HTTPS=true
.
.
.
.

The correct value is set when you just run the image manually, so where does it go when you run pack? And if I could find the script where it's set then perhaps I could run do the same in the build command.

[jmtd]

wrt which, we stopped including the which RPM and replaced uses of it in our bash scripts with type -p (see https://github.com/jboss-container-images/openjdk/pull/199) as part of a general effort to keep the image size as small as possible. As far as I can see, the only uses of which in this repo are in mvnw, which is a /bin/sh script. POSIX does not define type -p (which is a bash-ism) and type (without -p) behaves a little differently/unpredictably (dash here is an POSIX-sh-compliant non-bash shell):

$ dash -c "type mvn"
mvn is /usr/bin/mvn

One fix is just replacing /bin/sh in teh shebang with /bin/bash so long as all the images you care about will ship bash, and then which with type -p. But I guess mvnw is copied from some other canonical source and mvnw more generally might care about non-bash support. You probably don't want to diverge from upstream mvnw. In which case the fix is to install which inside the images, at some point in the process.

(looking at the other issues now)

[jmtd]

In both cases, the pack command is not passing through JAVA_HOME from the builder image. But in the working case, /etc/java/maven.conf exists, containing JAVA_HOME=/usr/lib/jvm/java-11-openjdk, and /usr/bin/mvn sources it. (The reason we remove the file is documented here).

One solution is just to install which: https://github.com/quarkusio/quarkus-buildpacks/pull/14, but there might be a better one.

[jmtd]

POSIX does not define type -p (which is a bash-ism) and type (without -p) behaves a little differently/unpredictably

On the other hand, command -v does what we need

$ command -v mvn
/usr/bin/mvn
$ echo $?
0
$ command -v doesnotexist
$ echo $?
1

[quintesse]

Thanks for all the info and the PR @jmtd !