logstash not starting after update to 8.13.0 from red hat package manager - files from bundled jdk missing after update

[walkerprog]

Logstash information:

Please include the following information:

1. Logstash version: 8.13.0
2. Logstash installation source: dnf package manager, baseurl=https://artifacts.elastic.co/packages/8.x/yum
3. How is Logstash being run: as system service

Plugins installed: logstash-codec-avro logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-http logstash-filter-json logstash-filter-kv logstash-filter-memcached logstash-filter-metrics logstash-filter-mutate logstash-filter-prune logstash-filter-ruby logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-uuid logstash-filter-xml logstash-input-azure_event_hubs logstash-input-beats └── logstash-input-elastic_agent (alias) logstash-input-couchdb_changes logstash-input-dead_letter_queue logstash-input-elastic_serverless_forwarder logstash-input-elasticsearch logstash-input-exec logstash-input-file logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-imap logstash-input-jms logstash-input-pipe logstash-input-redis logstash-input-snmp logstash-input-snmptrap logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter logstash-input-udp logstash-input-unix logstash-integration-aws ├── logstash-codec-cloudfront ├── logstash-codec-cloudtrail ├── logstash-input-cloudwatch ├── logstash-input-s3 ├── logstash-input-sqs ├── logstash-output-cloudwatch ├── logstash-output-s3 ├── logstash-output-sns └── logstash-output-sqs logstash-integration-elastic_enterprise_search ├── logstash-output-elastic_app_search └── logstash-output-elastic_workplace_search logstash-integration-jdbc ├── logstash-input-jdbc ├── logstash-filter-jdbc_streaming └── logstash-filter-jdbc_static logstash-integration-kafka ├── logstash-input-kafka └── logstash-output-kafka logstash-integration-logstash ├── logstash-input-logstash └── logstash-output-logstash logstash-integration-rabbitmq ├── logstash-input-rabbitmq └── logstash-output-rabbitmq logstash-output-csv logstash-output-elasticsearch logstash-output-email logstash-output-file logstash-output-graphite logstash-output-http logstash-output-lumberjack logstash-output-nagios logstash-output-null logstash-output-pipe logstash-output-redis logstash-output-stdout logstash-output-tcp logstash-output-udp logstash-output-webhdfs logstash-patterns-core [root@linux02 adm_linux02]# /usr/share/logstash/bin/logstash-plugin list --verbose Using bundled JDK: /usr/share/logstash/jdk logstash-codec-avro (3.4.1) logstash-codec-cef (6.2.7) logstash-codec-collectd (3.1.0) logstash-codec-dots (3.0.6) logstash-codec-edn (3.1.0) logstash-codec-edn_lines (3.1.0) logstash-codec-es_bulk (3.1.0) logstash-codec-fluent (3.4.2) logstash-codec-graphite (3.0.6) logstash-codec-json (3.1.1) logstash-codec-json_lines (3.1.0) logstash-codec-line (3.1.1) logstash-codec-msgpack (3.1.0) logstash-codec-multiline (3.1.1) logstash-codec-netflow (4.3.2) logstash-codec-plain (3.1.0) logstash-codec-rubydebug (3.1.0) logstash-filter-aggregate (2.10.0) logstash-filter-anonymize (3.0.7) logstash-filter-cidr (3.1.3) logstash-filter-clone (4.2.0) logstash-filter-csv (3.1.1) logstash-filter-date (3.1.15) logstash-filter-de_dot (1.0.4) logstash-filter-dissect (1.2.5) logstash-filter-dns (3.2.0) logstash-filter-drop (3.0.5) logstash-filter-elasticsearch (3.16.1) logstash-filter-fingerprint (3.4.3) logstash-filter-geoip (7.2.13) logstash-filter-grok (4.4.3) logstash-filter-http (1.5.1) logstash-filter-json (3.2.1) logstash-filter-kv (4.7.0) logstash-filter-memcached (1.2.0) logstash-filter-metrics (4.0.7) logstash-filter-mutate (3.5.8) logstash-filter-prune (3.0.4) logstash-filter-ruby (3.1.8) logstash-filter-sleep (3.0.7) logstash-filter-split (3.1.8) logstash-filter-syslog_pri (3.2.1) logstash-filter-throttle (4.0.4) logstash-filter-translate (3.4.2) logstash-filter-truncate (1.0.6) logstash-filter-urldecode (3.0.6) logstash-filter-useragent (3.3.5) logstash-filter-uuid (3.0.5) logstash-filter-xml (4.2.0) logstash-input-azure_event_hubs (1.4.5) logstash-input-beats (6.7.2) └── logstash-input-elastic_agent (alias) logstash-input-couchdb_changes (3.1.6) logstash-input-dead_letter_queue (2.0.0) logstash-input-elastic_serverless_forwarder (0.1.4) logstash-input-elasticsearch (4.19.1) logstash-input-exec (3.6.0) logstash-input-file (4.4.6) logstash-input-ganglia (3.1.4) logstash-input-gelf (3.3.2) logstash-input-generator (3.1.0) logstash-input-graphite (3.0.6) logstash-input-heartbeat (3.1.1) logstash-input-http (3.8.0) logstash-input-http_poller (5.5.1) logstash-input-imap (3.2.1) logstash-input-jms (3.2.2) logstash-input-pipe (3.1.0) logstash-input-redis (3.7.0) logstash-input-snmp (1.3.3) logstash-input-snmptrap (3.1.0) logstash-input-stdin (3.4.0) logstash-input-syslog (3.7.0) logstash-input-tcp (6.4.1) logstash-input-twitter (4.1.1) logstash-input-udp (3.5.0) logstash-input-unix (3.1.2) logstash-integration-aws (7.1.6) ├── logstash-codec-cloudfront ├── logstash-codec-cloudtrail ├── logstash-input-cloudwatch ├── logstash-input-s3 ├── logstash-input-sqs ├── logstash-output-cloudwatch ├── logstash-output-s3 ├── logstash-output-sns └── logstash-output-sqs logstash-integration-elastic_enterprise_search (3.0.0) ├── logstash-output-elastic_app_search └── logstash-output-elastic_workplace_search logstash-integration-jdbc (5.4.8) ├── logstash-input-jdbc ├── logstash-filter-jdbc_streaming └── logstash-filter-jdbc_static logstash-integration-kafka (11.3.4) ├── logstash-input-kafka └── logstash-output-kafka logstash-integration-logstash (1.0.2) ├── logstash-input-logstash └── logstash-output-logstash logstash-integration-rabbitmq (7.3.3) ├── logstash-input-rabbitmq └── logstash-output-rabbitmq logstash-output-csv (3.0.10) logstash-output-elasticsearch (11.22.2) logstash-output-email (4.1.3) logstash-output-file (4.3.0) logstash-output-graphite (3.1.6) logstash-output-http (5.6.0) logstash-output-lumberjack (3.1.9) logstash-output-nagios (3.0.6) logstash-output-null (3.0.5) logstash-output-pipe (3.0.6) logstash-output-redis (5.0.0) logstash-output-stdout (3.1.4) logstash-output-tcp (6.1.2) logstash-output-udp (3.2.0) logstash-output-webhdfs (3.1.0) logstash-patterns-core (4.3.4)

JVM (e.g. java -version):

should use the bundled JDK, no other java installed on server

OS version (uname -a if on a Unix-like system): RHEL 9.3 Linux servername.domain.local 5.14.0-362.24.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 15 07:18:13 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

after update from version 8.12.2 to 8.13.0 service will not start. logstash will not start when executing /usr/share/logstash/bin/logstash on console, too. Error message: Could not find java; set LS_JAVA_HOME or ensure java is in PATH. After a few debugging found the /usr/share/logstash/jdk folder containing the folder as the previos version (round about 87 folders), but NOT A SINGLE file. No wonder he looks for alternative java installed when there is no bundled anymore.

reverted back to 8.12.2 via dnf: files in /usr/share/logstash/jdk are back, logstash is running again as service AND on console with output "Using bundled JDK: /usr/share/logstash/jdk" at the beginning.

Updating to 8.13.0 again results to the behaviour described above - jdk-folder contains folders only, not a single file, logstash not starting, error-message: Could not find java; set LS_JAVA_HOME or ensure java is in PATH.

downgrading to 8.12.2 again and logstash works fine. can reproduce this in a circle

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

1. dnf clean all (be sure to be up to date)
2. dnf upgrade
3. confirm to update logstash from 8.12.2 to 8.13
4. after dnf has finished try to start logstash and verify the content of the bundled jdk-folder

Provide logs (if relevant):

Mar 26 19:31:49 servername logstash[51427]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH. Mar 26 19:31:49 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE Mar 26 19:31:49 servername systemd[1]: logstash.service: Failed with result 'exit-code'. Mar 26 19:31:49 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1. Mar 26 19:31:49 servername systemd[1]: Stopped logstash. Mar 26 19:31:49 servername systemd[1]: Started logstash. Mar 26 19:31:49 servername logstash[51438]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH. Mar 26 19:31:49 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE Mar 26 19:31:49 servername systemd[1]: logstash.service: Failed with result 'exit-code'. Mar 26 19:31:49 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 2. Mar 26 19:31:49 servername systemd[1]: Stopped logstash. Mar 26 19:31:49 servername systemd[1]: Started logstash. Mar 26 19:31:49 servername logstash[51447]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH. Mar 26 19:31:49 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE Mar 26 19:31:49 servername systemd[1]: logstash.service: Failed with result 'exit-code'. Mar 26 19:31:50 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 3. Mar 26 19:31:50 servername systemd[1]: Stopped logstash. Mar 26 19:31:50 servername systemd[1]: Started logstash. Mar 26 19:31:50 servername logstash[51456]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH. Mar 26 19:31:50 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE Mar 26 19:31:50 servername systemd[1]: logstash.service: Failed with result 'exit-code'. Mar 26 19:31:50 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 4. Mar 26 19:31:50 servername systemd[1]: Stopped logstash. Mar 26 19:31:50 servername systemd[1]: Started logstash. Mar 26 19:31:50 servername logstash[51465]: Could not find java; set LS_JAVA_HOME or ensure java is in PATH. Mar 26 19:31:50 servername systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE Mar 26 19:31:50 servername systemd[1]: logstash.service: Failed with result 'exit-code'. Mar 26 19:31:50 servername systemd[1]: logstash.service: Scheduled restart job, restart counter is at 5. Mar 26 19:31:50 servername systemd[1]: Stopped logstash. Mar 26 19:31:50 servername systemd[1]: logstash.service: Start request repeated too quickly. Mar 26 19:31:50 servername systemd[1]: logstash.service: Failed with result 'exit-code'. Mar 26 19:31:50 servername systemd[1]: Failed to start logstash.

[ ]# ls -lisha /usr/share/logstash/jdk/bin total 0 67110709 0 drwxr-xr-x. 2 root root 6 Mar 26 19:19 . 392218 0 drwxr-xr-x. 9 root root 92 Mar 26 19:19 .. []#

just as side node while installing 8.12.2 dnf downloads about 333MB while installing 8.13.0 dnf downloads about 210MB

either the code base REALLY shrinked with this version or there is missing a lot in the rpm-packages

[localhorst-org]

Same behavior under Ubuntu 22.04

Update from logstash 8.12.2 to 8.13.0

logstash install from https://artifacts.elastic.co/packages/8.x/apt stable main

no /usr/share/logstash/jdk/ folder

[gdsotirov]

Same with CentOS 8 Stream.

[jsvd]

Thanks for the report folks. it seems that the no-jdk package is being returned in the apt-get, as the artifacts for 8.13.0 in the download page here are ok, and include the jdk.

We're investigating what led to the debian package registry offering the no-jdk package by default instead of the one with JDK.

UPDATE: it looks like the no-jdk packages were incorrectly uploaded to the deb repository alongside the bundled jdk packages, and given they have the same priority, the no-jdk "all" architectures deb is being served first. UPDATE: users will experience the same with RPM packages.

[MakoWish]

Any workaround for now aside from downgrading? Thankfully we only updated our Development Logstash instance and ran into this.

[andsel]

You can download the deb or rpm for you distribution directly from https://www.elastic.co/downloads/logstash, for example https://artifacts.elastic.co/downloads/logstash/logstash-8.13.0-amd64.deb without passing through the package manager's repository, and install using the file you download.

[walkerprog]

can confirm the rpm-package from https://www.elastic.co/de/downloads/past-releases/logstash-8-13-0 works as expected installed with rpm -Uvh ./logstash-8.13.0-x86_64.rpm without problems, dnf no longer finds an update for logstash, files in jdk-folder are present and logstash-service is running

I really do hope dnf will recognize the next logstash-update and update from 8.13.0 to next version automatically

[ejgh-oe]

UPDATE: it looks like the no-jdk packages were incorrectly uploaded to the deb repository alongside the bundled jdk packages, and given they have the same priority, the no-jdk "all" architectures deb is being served first. UPDATE: users will experience the same with RPM packages.

Wrt. "deb": Also noticed that logstash went from amd64-package to "all" during the upgrade - your post explains alot... At least I now know I'm not alone with this...

[ejgh-oe]

Hi, Looks like the update is already out: did a apt update && apt upgrade on my Ubuntu 22.04 a couple of minutes ago and logstash:amd64 including java was installed again.

Thanks much for the quick fix.

[andsel]

I can confirm, that the deb package that's on repository, is the correct one. Verified with

$ sudo apt-get download --print-uris logstash
'https://artifacts.elastic.co/packages/8.x/apt/pool/main/l/logstash/logstash-8.13.0-amd64.deb' logstash_1%3a8.13.0-1_amd64.deb 412887318 SHA512:d575daa58f0390be7a03931efe93a11c1e00e6ed4e9c5c5f7a9c3aad455d006967b83269ddba531e0fc70e1b140b058ce10bb65831c7fe9aadd63046b9fd526f

[andsel]

@walkerprog does it fix for you?

[walkerprog]

had already fixed it with manually updating with downloaded rpm-file

quick test with downgrading to 8.12.2 and update with dnf to 8.13.0 was successful now the jdk-folder contains files (not only empty folders) and service starts and logstash is running although no other java is installed on the machine incoming syslog is processed as usual

Mar 29 15:03:05 servername logstash[2627299]: Using bundled JDK: /usr/share/logstash/jdk

thx for fixing and PLEASE make sure this never happens again :)

[roaksoax]

Closing the issue as it has now been solved.