Logstash crashes when running with JRE instead of JDK

[therth]

Logstash 8.13.2, unpacked from tar, started by systemd unit file Linux localhost.localdomain 5.14.0-362.24.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 15 07:18:13 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Issue: After removing the embedded JDK and using the provided JRE logstash crashes at startup.

java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)

When using the embedded JDK or setting LS_JAVA_HOME to another installation directory of JDK 17 all works fine.

Exception:

[2024-04-18T18:35:40,437][FATAL][org.logstash.Logstash    ][main] uncaught error (in thread Ruby-0-Thread-11: /opt/mcc-log-server/logstash-core/lib/logstash/java_pipeline.rb:289)
java.lang.NoClassDefFoundError: com/sun/source/tree/Tree

Full stacktrace added in logstash-plain.log.

logstash-plain.log

[tuck1s]

+1, also seen running under Ubuntu 22.04 LTS in Docker Desktop, then running Logstash interactively with the getting started example:

# /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ffffe960f72, pid=16314, tid=16348
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.11+9 (17.0.11+9) (build 17.0.11+9)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (17.0.11+9, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x73ef72]  G1ParScanThreadState::steal_and_trim_queue(GenericTaskQueueSet<OverflowTaskQueue<ScannerTask, (MEMFLAGS)5, 131072u>, (MEMFLAGS)5>*)+0x1d2
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# //hs_err_pid16314.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#
Aborted

hs_err_pid16314.log

Host is an Apple Silicon Mac, container OS is

uname -a
Linux 72fd9770ffa9 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Update: version logstash=1:8.7.1-1 seems to work for me a couple of times, then it's started crashing in the same way again.

[tuck1s]

Whereas on this VM

uname -a
Linux xx.xx.xx 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

latest version 1:8.13.4-1 works:

 /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2024-05-30 11:17:37.359 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.
[INFO ] 2024-05-30 11:17:37.376 [main] runner - Starting Logstash {"logstash.version"=>"8.13.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}
[INFO ] 2024-05-30 11:17:37.382 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[INFO ] 2024-05-30 11:17:37.387 [main] runner - Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[INFO ] 2024-05-30 11:17:37.397 [main] runner - Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[INFO ] 2024-05-30 11:17:37.415 [main] settings - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2024-05-30 11:17:37.418 [main] settings - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2024-05-30 11:17:37.737 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2024-05-30 11:17:37.751 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"b1ed0ec2-e832-47cc-bc11-c224b6953aeb", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2024-05-30 11:17:38.946 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2024-05-30 11:17:39.243 [Converge PipelineAction::Create<main>] Reflections - Reflections took 222 ms to scan 1 urls, producing 132 keys and 468 values
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/amazing_print-1.6.0/lib/amazing_print/formatter.rb:37: warning: previous definition of cast was here
[INFO ] 2024-05-30 11:17:39.615 [Converge PipelineAction::Create<main>] javapipeline - Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO ] 2024-05-30 11:17:39.679 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x74ff8ba7 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[INFO ] 2024-05-30 11:17:40.596 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.92}
[INFO ] 2024-05-30 11:17:40.634 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2024-05-30 11:17:40.644 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
The stdin plugin is now waiting for input: