search

logstash-plugins / logstash-filter-memcached

Memcached Filter plugin for Logstash
Apache License 2.0
4 stars 11 forks source link

Logstash crashes if logstash-filter-memcached can't reach the memcached server #34

Open jayaskren opened 2 months ago

jayaskren commented 2 months ago

Please include the following information:

  1. Logstash version : 8.8.2
  2. Logstash installation source: https://artifacts.elastic.co/downloads/logstash/logstash-8.8.2-arm64.deb
  3. How is Logstash being run: It is installed and run as a debian package
  4. How was the Logstash Plugin installed: We use the default installl of logstash

JVM (e.g. java -version): openjdk version "11.0.24" 2024-07-16 OpenJDK Runtime Environment (build 11.0.24+8-post-Ubuntu-1ubuntu322.04) OpenJDK 64-Bit Server VM (build 11.0.24+8-post-Ubuntu-1ubuntu322.04, mixed mode, sharing)

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

  1. JVM version (java -version)
  2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
  3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system): Linux logstash-c7-1 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: When the memcached filter can't reach the memcached server Logstash crashes. Ideally it would print a warning or something, but logstash itself would still keep running. This makes it really fragile in a production system.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

  1. Add the memcached filter to a logstash pipeline referencing a fake memcached server and the logstash server will crash

Provide logs (if relevant): Pipeline error Pipeline terminated Failed to execute action Logstash shut down. {"level":"ERROR","loggerName":"logstash.filters.memcached","timeMillis":1723116984952,"thread":"[main]-pipeline-manager","logEvent":{"message":"failed to connect to memcached","hosts":["logstash-memcached-stg-01-fouarx.serverless.usw2.cache.amazonaws.com:11212"],"options":{"expires_in":0},"message":"No server available"}}

jsvd commented 2 months ago

When logstash is started without automatic pipeline reloading, failures in a plugin's register phase will cause the pipeline to terminate, and so Logstash terminates too. This is because the pipeline was likely misconfigured and since auto reloading is off, it will require a process restart to load a "fixed" pipeline anyway.

If you enable pipeline reloading, you'll see the instance continues to be up even with a broken pipeline, and it keeps attempting to reload the pipeline until it's fixed.