search

WASdev / ci.docker

Build scripts for Docker images (Dockerfiles) and Docker related utilities for WebSphere Liberty.
Apache License 2.0
124 stars 121 forks source link

Exit code 22 during SCC layer creation #633

Open wraschke opened 2 weeks ago

wraschke commented 2 weeks ago

When Dockerfile code like

RUN if [ "$OPENJ9_SCC" = "true" ]; then populate_scc.sh;

is executed during a docker build, errors like the following occur intermittently:

#11 40.18 + /opt/ol/wlp/bin/server start
#11 ERROR: process "/bin/sh -c if [ \"$OPENJ9_SCC\" = \"true\" ]; then populate_scc.sh; fi     && rm -rf /output/messaging /output/resources/security /logs/* $WLP_OUTPUT_DIR/.classCache     && find /opt/ol/wlp/output ! -perm -g=rwx -print0 | xargs -0 -r chmod g+rwx" did not complete successfully: exit code: 22
------
 > [stage-1 4/4] RUN if [ "true" = "true" ]; then populate_scc.sh; fi     && rm -rf /output/messaging /output/resources/security /logs/* /opt/ol/wlp/output/.classCache     && find /opt/ol/wlp/output ! -perm -g=rwx -print0 | xargs -0 -r chmod g+rwx:
------
process "/bin/sh -c if [ \"$OPENJ9_SCC\" = \"true\" ]; then populate_scc.sh; fi     && rm -rf /output/messaging /output/resources/security /logs/* $WLP_OUTPUT_DIR/.classCache     && find /opt/ol/wlp/output ! -perm -g=rwx -print0 | xargs -0 -r chmod g+rwx" did not complete successfully: exit code: 22
An error occurred during container image building so the entire pipeline will fail.
[2024-08-26T19:39:03.093Z] 32.46 + /opt/ibm/wlp/bin/server start
[2024-08-26T19:39:03.093Z] ------
[2024-08-26T19:39:03.093Z] Dockerfile.ubi.openjdk17:49
[2024-08-26T19:39:03.093Z] --------------------
[2024-08-26T19:39:03.093Z]   48 |     # Create a new SCC layer
[2024-08-26T19:39:03.093Z]   49 | >>> RUN if [ "$OPENJ9_SCC" = "true" ]; then populate_scc.sh; fi \
[2024-08-26T19:39:03.093Z]   50 | >>>     && rm -rf /output/messaging /output/resources/security /logs/* $WLP_OUTPUT_DIR/.classCache \
[2024-08-26T19:39:03.093Z]   51 | >>>     && find /opt/ibm/wlp/output ! -perm -g=rwx -print0 | xargs -0 -r chmod g+rwx
[2024-08-26T19:39:03.093Z]   52 |     
[2024-08-26T19:39:03.093Z] --------------------
[2024-08-26T19:39:03.093Z] ERROR: failed to solve: process "/bin/sh -c if [ \"$OPENJ9_SCC\" = \"true\" ]; then populate_scc.sh; fi     && rm -rf /output/messaging /output/resources/security /logs/* $WLP_OUTPUT_DIR/.classCache     && find /opt/ibm/wlp/output ! -perm -g=rwx -print0 | xargs -0 -r chmod g+rwx" did not complete successfully: exit code: 22
[2024-08-26T10:19:18.159Z] 16.81 + /opt/ibm/wlp/bin/server stop
[2024-08-26T10:19:18.159Z] ------
[2024-08-26T10:19:18.159Z] Dockerfile.ubi.openjdk21:49
[2024-08-26T10:19:18.159Z] --------------------
[2024-08-26T10:19:18.159Z]   48 |     # Create a new SCC layer
[2024-08-26T10:19:18.159Z]   49 | >>> RUN if [ "$OPENJ9_SCC" = "true" ]; then populate_scc.sh; fi \
[2024-08-26T10:19:18.159Z]   50 | >>>     && rm -rf /output/messaging /output/resources/security /logs/* $WLP_OUTPUT_DIR/.classCache \
[2024-08-26T10:19:18.159Z]   51 | >>>     && find /opt/ibm/wlp/output ! -perm -g=rwx -print0 | xargs -0 -r chmod g+rwx
[2024-08-26T10:19:18.159Z]   52 |     
[2024-08-26T10:19:18.159Z] --------------------
[2024-08-26T10:19:18.159Z] ERROR: failed to solve: process "/bin/sh -c if [ \"$OPENJ9_SCC\" = \"true\" ]; then populate_scc.sh; fi     && rm -rf /output/messaging /output/resources/security /logs/* $WLP_OUTPUT_DIR/.classCache     && find /opt/ibm/wlp/output ! -perm -g=rwx -print0 | xargs -0 -r chmod g+rwx" did not complete successfully: exit code: 21

This has happened when building images for AMD and PowerPC (especially). I've noticed versions 24.0.0.6 and latest and with Java17 and Java21. There could be more combinations.

leochr commented 2 weeks ago

A similar error was observed during the build of getting-started sample application on PowerPC. This sample app invokes configure.sh which triggers SCC generation and server start/stop.

 ****** Building image: stg.icr.io/cp/olc-sample/open-liberty/samples/getting-started:latest-ppc64le
[2024-08-28T18:49:40.326Z] #0 building with "default" instance using docker driver
[2024-08-28T18:49:40.327Z] 
[2024-08-28T18:49:40.327Z] #1 [internal] load build definition from Dockerfile
[2024-08-28T18:49:40.327Z] #1 transferring dockerfile: 1.02kB done
[2024-08-28T18:49:40.327Z] #1 DONE 0.1s
[2024-08-28T18:49:40.327Z] 
[2024-08-28T18:49:40.327Z] #2 [internal] load metadata for icr.io/appcafe/open-liberty:kernel-slim-java11-openj9-ubi
[2024-08-28T18:49:40.327Z] #2 DONE 0.0s
[2024-08-28T18:49:40.327Z] 
[2024-08-28T18:49:40.327Z] #3 [internal] load .dockerignore
[2024-08-28T18:49:41.212Z] #3 transferring context:
[2024-08-28T18:49:41.212Z] #3 transferring context: 2B done
[2024-08-28T18:49:41.212Z] #3 DONE 1.2s
[2024-08-28T18:49:41.212Z] 
[2024-08-28T18:49:41.212Z] #4 [internal] load build context
[2024-08-28T18:49:41.212Z] #4 transferring context: 230.21kB 0.0s done
[2024-08-28T18:49:41.212Z] #4 DONE 0.2s
[2024-08-28T18:49:41.212Z] 
[2024-08-28T18:49:41.212Z] #5 [1/6] FROM icr.io/appcafe/open-liberty:kernel-slim-java11-openj9-ubi
[2024-08-28T18:49:42.099Z] #5 DONE 0.7s
[2024-08-28T18:49:42.099Z] 
[2024-08-28T18:49:42.099Z] #6 [2/6] COPY --chown=1001:0 src/main/liberty/config/ /config/
[2024-08-28T18:49:42.099Z] #6 DONE 0.1s
[2024-08-28T18:49:42.099Z] 
[2024-08-28T18:49:42.099Z] #7 [3/6] COPY --chown=1001:0 resources/ /output/resources/
[2024-08-28T18:49:42.099Z] #7 DONE 0.1s
[2024-08-28T18:49:42.099Z] 
[2024-08-28T18:49:42.099Z] #8 [4/6] RUN features.sh
[2024-08-28T18:49:42.980Z] #8 0.523 + SNIPPETS_SOURCE=/opt/ol/helpers/build/configuration_snippets
[2024-08-28T18:49:42.981Z] #8 0.523 + SNIPPETS_TARGET=/config/configDropins/overrides
[2024-08-28T18:49:42.981Z] #8 0.523 + SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
[2024-08-28T18:49:42.981Z] #8 0.523 + mkdir -p /config/configDropins/overrides
[2024-08-28T18:49:42.981Z] #8 0.536 + mkdir -p /config/configDropins/defaults
[2024-08-28T18:49:42.981Z] #8 0.541 + '[' -n '' ']'
[2024-08-28T18:49:42.981Z] #8 0.541 + '[' '' == client ']'
[2024-08-28T18:49:42.981Z] #8 0.541 + '[' '' == embedded ']'
[2024-08-28T18:49:42.981Z] #8 0.541 + [[ -n '' ]]
[2024-08-28T18:49:42.981Z] #8 0.541 + '[' '' == true ']'
[2024-08-28T18:49:42.981Z] #8 0.542 + '[' '' == true ']'
[2024-08-28T18:49:42.981Z] #8 0.542 + featureUtility installServerFeatures --acceptLicense defaultServer --noCache
[2024-08-28T18:50:26.159Z] #8 38.31 + find /opt/ol/wlp/lib /opt/ol/wlp/bin '!' -perm -g=rw -print0
[2024-08-28T18:50:26.159Z] #8 38.31 + xargs -0 -r chmod g+rw
[2024-08-28T18:50:26.159Z] #8 DONE 38.8s
[2024-08-28T18:50:26.159Z] 
[2024-08-28T18:50:26.159Z] #9 [5/6] COPY --chown=1001:0 target/*.war /config/apps/
[2024-08-28T18:50:26.159Z] #9 DONE 0.1s
[2024-08-28T18:50:26.159Z] 
[2024-08-28T18:50:26.159Z] #10 [6/6] RUN configure.sh && rm -rf /output/resources/security/
[2024-08-28T18:50:26.159Z] #10 0.511 + main
[2024-08-28T18:50:26.159Z] #10 0.511 + WLP_INSTALL_DIR=/opt/ol/wlp
[2024-08-28T18:50:26.159Z] #10 0.512 + SHARED_CONFIG_DIR=/opt/ol/wlp/usr/shared/config
[2024-08-28T18:50:26.159Z] #10 0.512 + SHARED_RESOURCE_DIR=/opt/ol/wlp/usr/shared/resources
[2024-08-28T18:50:26.159Z] #10 0.512 + SNIPPETS_SOURCE=/opt/ol/helpers/build/configuration_snippets
[2024-08-28T18:50:26.159Z] #10 0.512 + SNIPPETS_TARGET=/config/configDropins/overrides
[2024-08-28T18:50:26.159Z] #10 0.512 + SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
[2024-08-28T18:50:26.159Z] #10 0.512 + mkdir -p /config/configDropins/overrides
[2024-08-28T18:50:26.159Z] #10 0.517 + mkdir -p /config/configDropins/defaults
[2024-08-28T18:50:26.159Z] #10 0.522 + [[ -n '' ]]
[2024-08-28T18:50:26.159Z] #10 0.522 + '[' '' == client ']'
[2024-08-28T18:50:26.159Z] #10 0.522 + '[' '' == embedded ']'
[2024-08-28T18:50:26.159Z] #10 0.522 + keystorePath=/config/configDropins/defaults/keystore.xml
[2024-08-28T18:50:26.159Z] #10 0.522 + '[' '' '!=' false ']'
[2024-08-28T18:50:26.159Z] #10 0.522 + '[' '' '!=' false ']'
[2024-08-28T18:50:26.159Z] #10 0.522 + '[' '!' -e /config/configDropins/defaults/keystore.xml ']'
[2024-08-28T18:50:26.159Z] #10 0.523 ++ openssl rand -base64 32
[2024-08-28T18:50:26.159Z] #10 0.531 + export KEYSTOREPWD=BtGmoZ4Itfauj5Am+apvxingtyIRGFnNuvQr+wQAlr8=
[2024-08-28T18:50:26.159Z] #10 0.531 + KEYSTOREPWD=BtGmoZ4Itfauj5Am+apvxingtyIRGFnNuvQr+wQAlr8=
[2024-08-28T18:50:26.159Z] #10 0.531 + sed 's|REPLACE|BtGmoZ4Itfauj5Am+apvxingtyIRGFnNuvQr+wQAlr8=|g' /opt/ol/helpers/build/configuration_snippets/keystore.xml
[2024-08-28T18:50:26.159Z] #10 0.536 + chmod g+w /config/configDropins/defaults/keystore.xml
[2024-08-28T18:50:26.159Z] #10 0.541 + [[ -n '' ]]
[2024-08-28T18:50:26.159Z] #10 0.543 + find /opt/ol/fixes -type f -name '*.jar' -print0
[2024-08-28T18:50:26.159Z] #10 0.543 + sort -z
[2024-08-28T18:50:26.159Z] #10 0.543 + xargs -0 -n 1 -r -I '{}' java -jar '{}' --installLocation /opt/ol/wlp
[2024-08-28T18:50:26.159Z] #10 0.547 + touch /config/server.xml
[2024-08-28T18:50:26.159Z] #10 0.554 + '[' true == true ']'
[2024-08-28T18:50:26.159Z] #10 0.554 + cmd='populate_scc.sh -i 1'
[2024-08-28T18:50:26.159Z] #10 0.554 + '[' '' == false ']'
[2024-08-28T18:50:26.159Z] #10 0.554 + '[' '!' '' = '' ']'
[2024-08-28T18:50:26.159Z] #10 0.554 + '[' '' = false ']'
[2024-08-28T18:50:26.159Z] #10 0.554 + '[' '!' '' = '' ']'
[2024-08-28T18:50:26.159Z] #10 0.554 + '[' '' = false ']'
[2024-08-28T18:50:26.159Z] #10 0.554 + '[' '!' '' = '' ']'
[2024-08-28T18:50:26.160Z] #10 0.554 + eval populate_scc.sh -i 1
[2024-08-28T18:50:26.160Z] #10 0.554 ++ populate_scc.sh -i 1
[2024-08-28T18:50:26.160Z] #10 0.559 + SCC_SIZE=80m
[2024-08-28T18:50:26.160Z] #10 0.559 + ITERATIONS=2
[2024-08-28T18:50:26.160Z] #10 0.559 + TRIM_SCC=yes
[2024-08-28T18:50:26.160Z] #10 0.559 + WARM_ENDPOINT=true
[2024-08-28T18:50:26.160Z] #10 0.559 + WARM_ENDPOINT_URL=localhost:9080/
[2024-08-28T18:50:26.160Z] #10 0.559 + WARM_OPENAPI_ENDPOINT=true
[2024-08-28T18:50:26.160Z] #10 0.560 + WARM_OPENAPI_ENDPOINT_URL=localhost:9080/openapi
[2024-08-28T18:50:26.160Z] #10 0.560 + [[ -d /opt/java/.scc ]]
[2024-08-28T18:50:26.160Z] #10 0.561 ++ stat -L -c %a /opt/java/.scc
[2024-08-28T18:50:26.160Z] #10 0.561 ++ cut -c 1,2
[2024-08-28T18:50:26.160Z] #10 0.567 + [[ 77 == \7\7 ]]
[2024-08-28T18:50:26.160Z] #10 0.567 + SCC=-Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc
[2024-08-28T18:50:26.160Z] #10 0.568 + export 'OPENJ9_JAVA_OPTIONS=-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
[2024-08-28T18:50:26.160Z] #10 0.568 + OPENJ9_JAVA_OPTIONS='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
[2024-08-28T18:50:26.160Z] #10 0.568 + export 'IBM_JAVA_OPTIONS=-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
[2024-08-28T18:50:26.160Z] #10 0.568 + IBM_JAVA_OPTIONS='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
[2024-08-28T18:50:26.160Z] #10 0.568 + CREATE_LAYER='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,createLayer,groupAccess'
[2024-08-28T18:50:26.160Z] #10 0.569 + DESTROY_LAYER='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,destroy'
[2024-08-28T18:50:26.160Z] #10 0.569 + PRINT_LAYER_STATS='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,printTopLayerStats'
[2024-08-28T18:50:26.160Z] #10 0.570 + getopts :i:s:u:o:tdhwcml OPT
[2024-08-28T18:50:26.160Z] #10 0.570 + case "$OPT" in
[2024-08-28T18:50:26.160Z] #10 0.570 + ITERATIONS=1
[2024-08-28T18:50:26.160Z] #10 0.570 + getopts :i:s:u:o:tdhwcml OPT
[2024-08-28T18:50:26.160Z] #10 0.571 ++ umask
[2024-08-28T18:50:26.160Z] #10 0.572 + OLD_UMASK=0022
[2024-08-28T18:50:26.160Z] #10 0.572 + umask 002
[2024-08-28T18:50:26.160Z] #10 0.572 + java -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,createLayer,groupAccess -Xscmx80m -version
[2024-08-28T18:50:26.161Z] #10 0.870 openjdk version "11.0.24" 2024-07-16
[2024-08-28T18:50:26.161Z] #10 0.871 IBM Semeru Runtime Open Edition 11.0.24.0 (build 11.0.24+8)
[2024-08-28T18:50:26.161Z] #10 0.871 Eclipse OpenJ9 VM 11.0.24.0 (build openj9-0.46.0, JRE 11 Linux ppc64le-64-Bit Compressed References 20240802_1020 (JIT enabled, AOT enabled)
[2024-08-28T18:50:26.161Z] #10 0.871 OpenJ9   - 1a6f6128aa
[2024-08-28T18:50:26.161Z] #10 0.871 OMR      - 840a9adba
[2024-08-28T18:50:26.161Z] #10 0.871 JCL      - 30b62e68b8 based on jdk-11.0.24+8)
[2024-08-28T18:50:26.161Z] #10 0.886 + '[' yes == yes ']'
[2024-08-28T18:50:26.161Z] #10 0.887 + echo 'Calculating SCC layer upper bound, starting with initial size 80m.'
[2024-08-28T18:50:26.161Z] #10 0.887 + /opt/ol/wlp/bin/server start
[2024-08-28T18:50:38.892Z] #10 17.56 + '[' true == true ']'
[2024-08-28T18:50:38.892Z] #10 17.57 + curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/
[2024-08-28T18:50:39.791Z] #10 18.71 + '[' true == true ']'
[2024-08-28T18:50:39.791Z] #10 18.71 + curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/openapi
[2024-08-28T18:50:39.791Z] #10 18.74 + echo 'localhost:9080/openapi call failed, continuing'
[2024-08-28T18:50:39.791Z] #10 18.74 + /opt/ol/wlp/bin/server stop
[2024-08-28T18:50:43.639Z] #10 21.99 ++ awk '/^Cache is [0-9.]*% .*full/ {print substr($3, 1, length($3)-1)}'
[2024-08-28T18:50:43.639Z] #10 22.14 + FULL=40
[2024-08-28T18:50:43.639Z] #10 22.14 + echo 'SCC layer is 40% full. Destroying layer.'
[2024-08-28T18:50:43.639Z] #10 22.14 + java -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,destroy
[2024-08-28T18:50:43.639Z] #10 22.19 JVMSHRC806I Compressed references persistent shared cache "openj9_system_scc" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
[2024-08-28T18:50:43.639Z] #10 22.19 + true
[2024-08-28T18:50:43.639Z] #10 22.19 + SCC_SIZE=80
[2024-08-28T18:50:43.639Z] #10 22.19 ++ awk 'BEGIN {print int(80 * 40 / 100.0 + 0.5)}'
[2024-08-28T18:50:43.639Z] #10 22.19 + SCC_SIZE=32
[2024-08-28T18:50:43.639Z] #10 22.19 + '[' 32 -eq 0 ']'
[2024-08-28T18:50:43.639Z] #10 22.19 + SCC_SIZE=32m
[2024-08-28T18:50:43.639Z] #10 22.19 + echo 'Re-creating layer with size 32m.'
[2024-08-28T18:50:43.639Z] #10 22.19 + java -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,createLayer,groupAccess -Xscmx32m -version
[2024-08-28T18:50:43.639Z] #10 22.48 openjdk version "11.0.24" 2024-07-16
[2024-08-28T18:50:43.639Z] #10 22.48 IBM Semeru Runtime Open Edition 11.0.24.0 (build 11.0.24+8)
[2024-08-28T18:50:43.639Z] #10 22.48 Eclipse OpenJ9 VM 11.0.24.0 (build openj9-0.46.0, JRE 11 Linux ppc64le-64-Bit Compressed References 20240802_1020 (JIT enabled, AOT enabled)
[2024-08-28T18:50:43.639Z] #10 22.48 OpenJ9   - 1a6f6128aa
[2024-08-28T18:50:43.639Z] #10 22.48 OMR      - 840a9adba
[2024-08-28T18:50:43.639Z] #10 22.48 JCL      - 30b62e68b8 based on jdk-11.0.24+8)
[2024-08-28T18:50:43.639Z] #10 22.50 + (( i=0 ))
[2024-08-28T18:50:43.639Z] #10 22.50 + (( i<1 ))
[2024-08-28T18:50:43.639Z] #10 22.50 + /opt/ol/wlp/bin/server start
[2024-08-28T18:51:03.868Z] #10 ERROR: process "/bin/sh -c configure.sh && rm -rf /output/resources/security/" did not complete successfully: exit code: 22
[2024-08-28T18:51:03.868Z] ------
[2024-08-28T18:51:03.868Z]  > [6/6] RUN configure.sh && rm -rf /output/resources/security/:
[2024-08-28T18:51:03.868Z] 22.19 + java -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,createLayer,groupAccess -Xscmx32m -version
[2024-08-28T18:51:03.868Z] 22.48 openjdk version "11.0.24" 2024-07-16
[2024-08-28T18:51:03.868Z] 22.48 IBM Semeru Runtime Open Edition 11.0.24.0 (build 11.0.24+8)
[2024-08-28T18:51:03.868Z] 22.48 Eclipse OpenJ9 VM 11.0.24.0 (build openj9-0.46.0, JRE 11 Linux ppc64le-64-Bit Compressed References 20240802_1020 (JIT enabled, AOT enabled)
[2024-08-28T18:51:03.868Z] 22.48 OpenJ9   - 1a6f6128aa
[2024-08-28T18:51:03.868Z] 22.48 OMR      - 840a9adba
[2024-08-28T18:51:03.868Z] 22.48 JCL      - 30b62e68b8 based on jdk-11.0.24+8)
[2024-08-28T18:51:03.868Z] 22.50 + (( i=0 ))
[2024-08-28T18:51:03.868Z] 22.50 + (( i<1 ))
[2024-08-28T18:51:03.868Z] 22.50 + /opt/ol/wlp/bin/server start
[2024-08-28T18:51:03.868Z] ------
[2024-08-28T18:51:03.868Z] Dockerfile:25
[2024-08-28T18:51:03.868Z] --------------------
[2024-08-28T18:51:03.868Z]   23 |     COPY --chown=1001:0 target/*.war /config/apps/
[2024-08-28T18:51:03.868Z]   24 |     
[2024-08-28T18:51:03.868Z]   25 | >>> RUN configure.sh && rm -rf /output/resources/security/
[2024-08-28T18:51:03.868Z]   26 |     
[2024-08-28T18:51:03.868Z] --------------------
[2024-08-28T18:51:03.868Z] ERROR: failed to solve: process "/bin/sh -c configure.sh && rm -rf /output/resources/security/" did not complete successfully: exit code: 22
[2024-08-28T18:51:03.868Z] make: *** [Makefile:11: build-app-pipeline] Error 1
halim-lee commented 2 weeks ago

Reproduced error logs on Jenkins s390x build with Open Liberty & Java 17:

[2024-08-30T15:23:08.136Z] #13 37.61 + /opt/ol/wlp/bin/server start
[2024-08-30T15:23:19.189Z] #13 48.10 + ec=22
[2024-08-30T15:23:19.189Z] #13 48.10 + echo '==> start failed'
[2024-08-30T15:23:19.189Z] #13 48.11 ==> start failed
...
...
[2024-08-30T15:23:19.190Z] #13 48.12 ********************************************************************************
[2024-08-30T15:23:19.190Z] #13 48.12 product = Open Liberty 24.0.0.8 (wlp-1.0.92.cl240820240729-1903)
[2024-08-30T15:23:19.190Z] #13 48.12 wlp.install.dir = /opt/ol/wlp/
[2024-08-30T15:23:19.190Z] #13 48.12 server.output.dir = /opt/ol/wlp/output/defaultServer/
[2024-08-30T15:23:19.190Z] #13 48.12 java.home = /opt/java/openjdk
[2024-08-30T15:23:19.190Z] #13 48.12 java.version = 17.0.12
[2024-08-30T15:23:19.190Z] #13 48.12 java.runtime = IBM Semeru Runtime Open Edition (17.0.12+7)
[2024-08-30T15:23:19.190Z] #13 48.12 os = Linux (5.4.0-181-generic; s390x) (en_US)
[2024-08-30T15:23:19.190Z] #13 48.12 process = 271@buildkitsandbox
[2024-08-30T15:23:19.190Z] #13 48.12 Classpath = /opt/ol/wlp/bin/tools/ws-server.jar
[2024-08-30T15:23:19.190Z] #13 48.12 Java Library path = /opt/java/openjdk/lib/default:/opt/java/openjdk/lib:/usr/lib64:/usr/lib
[2024-08-30T15:23:19.190Z] #13 48.12 ********************************************************************************
[2024-08-30T15:23:19.190Z] #13 48.12 [8/30/24, 15:23:08:995 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager           A CWWKE0001I: The server defaultServer has been launched.
[2024-08-30T15:23:19.190Z] #13 48.12 [8/30/24, 15:23:10:237 UTC] 00000022 com.ibm.ws.config.xml.internal.ServerXMLConfiguration        A CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[2024-08-30T15:23:19.191Z] #13 48.12 [8/30/24, 15:23:10:312 UTC] 00000022 com.ibm.ws.config.xml.internal.ServerXMLConfiguration        A CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[2024-08-30T15:23:19.191Z] #13 48.12 [8/30/24, 15:23:10:610 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager           I CWWKE0002I: The kernel started after 2.744 seconds
[2024-08-30T15:23:19.191Z] #13 48.12 [8/30/24, 15:23:10:695 UTC] 0000002d com.ibm.ws.kernel.feature.internal.FeatureManager            I CWWKF0007I: Feature update started.
[2024-08-30T15:23:19.191Z] #13 48.12 [8/30/24, 15:23:10:958 UTC] 00000021 com.ibm.ws.security.ready.internal.SecurityReadyServiceImpl  I CWWKS0007I: The security service is starting...
[2024-08-30T15:23:19.191Z] #13 48.12 + cat /logs/console.log
[2024-08-30T15:23:19.191Z] #13 48.12 Launching defaultServer (Open Liberty 24.0.0.8/wlp-1.0.92.cl240820240729-1903) on Eclipse OpenJ9 VM, version 17.0.12+7 (en_US)
[2024-08-30T15:23:19.191Z] #13 48.12 [AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[2024-08-30T15:23:19.191Z] #13 48.12 [AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[2024-08-30T15:23:19.191Z] #13 48.12 [AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[2024-08-30T15:23:19.191Z] #13 48.12 Unhandled exception
[2024-08-30T15:23:19.191Z] #13 48.12 Type=Segmentation error vmState=0x00080001
[2024-08-30T15:23:19.191Z] #13 48.12 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2024-08-30T15:23:19.191Z] #13 48.12 Handler1=000003FF88F4A798 Handler2=000003FF88E31A60 InaccessibleAddress=000003FFA4411000
[2024-08-30T15:23:19.191Z] #13 48.12 gpr0=000003FF84411550 gpr1=000003FF84411490 gpr2=000000002000003C gpr3=00000000FFFFFFFE
[2024-08-30T15:23:19.191Z] #13 48.12 gpr4=0000000000000000 gpr5=000003FF842DEFA0 gpr6=000003FF842F8530 gpr7=000003FF8441B348
[2024-08-30T15:23:19.191Z] #13 48.12 gpr8=0000000000000B07 gpr9=000003FF82A1E2B0 gpr10=000003FF84412057 gpr11=000003FF842DEF90
[2024-08-30T15:23:19.191Z] #13 48.12 gpr12=000003FF677B8858 gpr13=0000000001681400 gpr14=000003FF829F3846 gpr15=000003FF888FDF38
[2024-08-30T15:23:19.191Z] #13 48.12 psw=000003FF829F6AB2 mask=0705200180000000 fpc=00080000 bea=000003FF829F6D1C
[2024-08-30T15:23:19.191Z] #13 48.12 fpr0=3d61b81900000000 (f: 0.000000, d: 5.036080e-13)
[2024-08-30T15:23:19.191Z] #13 48.12 fpr1=000003ff8989f968 (f: 2307520768.000000, d: 2.171942e-311)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr2=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr3=0000000000000007 (f: 7.000000, d: 3.458460e-323)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr4=000003ff84567cde (f: 2220260608.000000, d: 2.171899e-311)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr5=000003ff84567d08 (f: 2220260608.000000, d: 2.171899e-311)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr7=6489e0d949b4555a (f: 1236555136.000000, d: 2.048158e+176)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr9=000002aa1a59ad80 (f: 442084736.000000, d: 1.447420e-311)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr10=000003ff5801ca88 (f: 1476512384.000000, d: 2.171531e-311)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr11=00000000ccaa009e (f: 3433693440.000000, d: 1.696470e-314)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr12=000003ff888ff910 (f: 2291136768.000000, d: 2.171934e-311)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr14=0000000000041000 (f: 266240.000000, d: 1.315400e-318)
[2024-08-30T15:23:19.192Z] #13 48.12 fpr15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-30T15:23:19.192Z] #13 48.12 Module=/opt/java/openjdk/lib/default/libjclse29.so
[2024-08-30T15:23:19.192Z] #13 48.12 Module_base_address=000003FF82980000
[2024-08-30T15:23:19.192Z] #13 48.12 Target=2_90_20240716_774 (Linux 5.4.0-181-generic)
[2024-08-30T15:23:19.192Z] #13 48.12 CPU=s390x (2 logical CPUs) (0x1f61b6000 RAM)
[2024-08-30T15:23:19.192Z] #13 48.12 ----------- Stack Backtrace -----------
[2024-08-30T15:23:19.192Z] #13 48.12 pool_newElement+0x112 (0x000003FF829F6AB2 [libjclse29.so+0x76ab2])
[2024-08-30T15:23:19.192Z] #13 48.12 hashTableAddNodeInList+0xb6 (0x000003FF829F3846 [libjclse29.so+0x73846])
[2024-08-30T15:23:19.192Z] #13 48.12 getCachedString+0x164 (0x000003FF829C19D4 [libjclse29.so+0x419d4])
[2024-08-30T15:23:19.192Z] #13 48.12 getCachedURL+0x16a (0x000003FF829C2F32 [libjclse29.so+0x42f32])
[2024-08-30T15:23:19.192Z] #13 48.12 Java_com_ibm_oti_shared_SharedClassURLHelperImpl_findSharedClassImpl3+0x41e (0x000003FF829C44F6 [libjclse29.so+0x444f6])
[2024-08-30T15:23:19.192Z] #13 48.12  (0x000003FF677B650C [<unknown>+0x0])
[2024-08-30T15:23:19.192Z] #13 48.12 ---------------------------------------
tjwatson commented 2 weeks ago

We have seen other crashes similar to this:

https://github.com/OpenLiberty/open-liberty/issues/29486 https://github.com/eclipse-openj9/openj9/issues/20012

We need some help from the Open J9 team to see if this will be fixed in a new release of Semeru Java.

leochr commented 1 week ago

Java team confirmed that this failure is due to https://github.com/eclipse-openj9/openj9/issues/20012. We are waiting for the Semeru container images to be updated by Java team to include the fix. Once they do, the Liberty images will be refreshed to include the fix. We'll post an update to this issue once we confirm that the fix is included in Liberty images.