search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
91 stars 3.5k forks source link

logstash 7.5.1 failed start in "java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;" #12133

Open thanh-lam opened 4 years ago

thanh-lam commented 4 years ago
TheVastyDeep commented 4 years ago

Error occurs at this call to position. Possibly similar to this issue, where the expected return type of the position method for ByteBuffer was different for different Java versions, so a mismatch between the compile and runtime versions of Java caused the issue.

thanh-lam commented 4 years ago

Hello, Thanks much for looking into the problem! I tried to run logstash on command line instead of using systemctl start logstash and it started. The cli is something like:

bin/logstash -f /etc/logstash/conf.d/logstash.conf --path.logs /var/log/logstash/clilog --path.settings /etc/logstash

As far as I can see, this cli is not much different from the command in the systemd service:

# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

Do you see anything here that could cause logstash fail to start like from the command line? Thanks again! I'm hopeful that this is leading to some problem that we've missed.

TheVastyDeep commented 4 years ago

When logstash is running, run ps and check logstash process. ps will show you which java it is running and also the classpath. Maybe one or other is different when you run it on the command line.

thanh-lam commented 4 years ago

Thanks again for the diagnosis on the java versions mismatch. That's very helpful. That has been my suspicion all along but it was too hard to find the source of the error. I think I did look for the logstash process with the ps command many times but just missed the part on which java is running logstash (It's a long command... probably with the CLASSPATH as you pointed out).

What did help me found and fix the problem is the output from the systemctl status logstash command. This command also printed out "live" messages when starting logstash. It's hard to catch all the messages.... I've tried running it so many times and it finally gave me the following clue:

# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-07-24 10:22:46 EDT; 2s ago
 Main PID: 1423 (java)
    Tasks: 72 (limit: 117964)
   Memory: 553.8M
   CGroup: /system.slice/logstash.service
           └─1423 /bin/java -Xms64g -Xmx64g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+Use>

With that "/bin/java ..." info, I found out /bin/java is a symbolic link to the java-1.8 version! So, systemctl did not use the user's $JAVA_HOME setting that point to /usr/lib/jvm/java-13. How does it decide which version of JVM to use?

Back to the file logstash.service for logstash that I posted earlier, systemctl uses the file /etc/default/logstash for its environment: EnvironmentFile=-/etc/default/logstash. But, no mention about java or JVM. Here's the file content:

# cat logstash
LS_HOME="/usr/share/logstash"
LS_SETTINGS_DIR="/etc/logstash"
LS_PIDFILE="/var/run/logstash/logstash.pid"
LS_USER="logstash"
LS_GROUP="logstash"
LS_GC_LOG_FILE="/var/log/logstash/gc.log"
LS_OPEN_FILES="16384"
LS_NICE="19"
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"

I gave the command systemctl start logstash one more try, with the line JAVA_HOME=/usr/lib/jvm/java-13 and that works.

My follow on question is: When or which process generates the file /etc/default/logstash? Is it at the time of installing logstash? This is a simple thing to fix the multiple java versions mismatch problem. It's already required that user set $JAVA_HOME. So it's just a matter of adding that line into /etc/default/logstash.

I would also suggest adding documentation about this "multiple" java versions mismatching problem. That will help preventing the problem or help debugging the problem.

thanh-lam commented 4 years ago

Last time I had to close the issue because it didn't let me add comment (the Comment button was grey out). I'm reopening this because I found a fix and also documentation should be added about this problem.