search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.21k stars 3.5k forks source link

AppInfo mbean errors since update to 8.11.4, logstash now terminates after upgrading to 8.12.0 #15821

Open rsommer opened 9 months ago

rsommer commented 9 months ago

Logstash information:

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version)

    Using bundled JDK: /usr/share/logstash/jdk
logstash 8.11.4

  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) debs from elastic.co

  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) Run as systemd-service

Plugins installed: (bin/logstash-plugin list --verbose)

Using bundled JDK: /usr/share/logstash/jdk
logstash-codec-avro (3.4.1)
logstash-codec-cef (6.2.7)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.2)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.3.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.7)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.2.0)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.16.1)
logstash-filter-fingerprint (3.4.3)
logstash-filter-geoip (7.2.13)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.5.0)
logstash-filter-json (3.2.1)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.2.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.8)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.2.0)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.4.2)
logstash-filter-truncate (1.0.6)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.5)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.2.0)
logstash-input-azure_event_hubs (1.4.5)
logstash-input-beats (6.7.2)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (2.0.0)
logstash-input-elastic_serverless_forwarder (0.1.4)
logstash-input-elasticsearch (4.18.0)
logstash-input-exec (3.6.0)
logstash-input-file (4.4.6)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.7.3)
logstash-input-http_poller (5.5.1)
logstash-input-imap (3.2.1)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-snmp (1.3.3)
logstash-input-snmptrap (3.1.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.7.0)
logstash-input-tcp (6.4.1)
logstash-input-twitter (4.1.1)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-aws (7.1.6)
 ├── logstash-codec-cloudfront
 ├── logstash-codec-cloudtrail
 ├── logstash-input-cloudwatch
 ├── logstash-input-s3
 ├── logstash-input-sqs
 ├── logstash-output-cloudwatch
 ├── logstash-output-s3
 ├── logstash-output-sns
 └── logstash-output-sqs
logstash-integration-elastic_enterprise_search (2.2.1)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.4.6)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (11.3.2)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-logstash (0.0.5)
 ├── logstash-input-logstash
 └── logstash-output-logstash
logstash-integration-rabbitmq (7.3.3)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-csv (3.0.10)
logstash-output-elasticsearch (11.19.0)
logstash-output-email (4.1.3)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.6.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.1.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.1.0)
logstash-patterns-core (4.3.4)

JVM (e.g. java -version):

openjdk 17.0.9 2023-10-17
OpenJDK Runtime Environment Temurin-17.0.9+9 (build 17.0.9+9)
OpenJDK 64-Bit Server VM Temurin-17.0.9+9 (build 17.0.9+9, mixed mode, sharing)

OS version (uname -a if on a Unix-like system): Linux logstash01 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: Starting with the update to 8.11.4, we encounter errors like this:

logstash[1701512]: [WARN ] 2024-01-15 08:06:57.013 [[main]<kafka] AppInfoParser - Error registering AppInfo mbean
logstash[1701512]: javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=logstash-0

This went unnoticed for the 8.11.4 release, because the logstash instances start anyway and behave otherwise correct. Maybe this was present earlier, but we do not keep the logstash logs that long.

After upgrading to 8.12.0 today (which went fine for all other elastic components like elasticsearch and kibana), this exception leads to restart loops within the logstash services.

A similar problem was discussed in https://discuss.elastic.co/t/logstash-group-imbalance-due-consumer-thread/141075 - the hint to just ignore this output does not "work" anymore, because now the process is stopped when encountering this error.

Provide logs (if relevant):

Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [WARN ] 2024-01-18 11:15:44.995 [[main]<kafka] AppInfoParser - Error registering AppInfo mbean
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=logstash-0
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1865) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:960) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:895) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:523) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:64) ~[kafka-clients-3.4.1.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:821) ~[kafka-clients-3.4.1.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:665) ~[kafka-clients-3.4.1.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646) ~[kafka-clients-3.4.1.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:626) ~[kafka-clients-3.4.1.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.javasupport.JavaConstructor.newInstanceDirect(JavaConstructor.java:165) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.java.invokers.ConstructorInvoker.call(ConstructorInvoker.java:85) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.java.invokers.ConstructorInvoker.call(ConstructorInvoker.java:174) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:257) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.java.proxies.ConcreteJavaProxy$InitializeMethod.call(ConcreteJavaProxy.java:93) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:257) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyClass.newInstance(RubyClass.java:904) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneOrNBlock.call(JavaMethod.java:355) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.java.proxies.ConcreteJavaProxy$NewMethod.call(ConcreteJavaProxy.java:188) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:318) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:680) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at usr.share.logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_integration_minus_kafka_minus_11_dot_3_dot_3_minus_java.lib.logstash.inputs.kafka.RUBY$block$run$1(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-integration-kafka-11.3.3-java/lib/logstash/inputs/kafka.rb:297) ~[?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:151) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.BlockBody.yield(BlockBody.java:106) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.Block.yield(Block.java:189) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerable$19.call(RubyEnumerable.java:851) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.CallBlock19.doYield(CallBlock19.java:111) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.BlockBody.yield(BlockBody.java:116) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.Block.yieldValues(Block.java:238) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerator.lambda$__each__$0(RubyEnumerator.java:329) ~[jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.BlockCallback.call(BlockCallback.java:58) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.CallBlock.doYield(CallBlock.java:97) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.BlockBody.yield(BlockBody.java:108) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.Block.yield(Block.java:189) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyFixnum.times(RubyFixnum.java:308) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyInteger$INVOKER$i$0$0$times.call(RubyInteger$INVOKER$i$0$0$times.gen) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:561) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:522) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyClass.finvoke(RubyClass.java:510) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.Helpers.invoke(Helpers.java:704) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:388) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerator.__each__(RubyEnumerator.java:327) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerator.each(RubyEnumerator.java:322) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerator$INVOKER$i$each.call(RubyEnumerator$INVOKER$i$each.gen) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:208) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerable.collectCommon(RubyEnumerable.java:843) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerable.map(RubyEnumerable.java:835) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyEnumerable$INVOKER$s$0$0$map.call(RubyEnumerable$INVOKER$s$0$0$map.gen) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:561) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:725) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at usr.share.logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_integration_minus_kafka_minus_11_dot_3_dot_3_minus_java.lib.logstash.inputs.kafka.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-integration-kafka-11.3.3-java/lib/logstash/inputs/kafka.rb:295) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at usr.share.logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_integration_minus_kafka_minus_11_dot_3_dot_3_minus_java.lib.logstash.inputs.kafka.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-integration-kafka-11.3.3-java/lib/logstash/inputs/kafka.rb:294) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:414) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:409) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:680) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.runtime.Block.call(Block.java:144) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.RubyProc.call(RubyProc.java:352) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111) [jruby.jar:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]:         at java.lang.Thread.run(Thread.java:840) [?:?]
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.028 [[main]<kafka] KafkaConsumer - [Consumer clientId=logstash-0, groupId=logstash-syslog-staging-dc01] Subscribed to topic(s): syslog
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.030 [[main]<kafka] KafkaConsumer - [Consumer clientId=logstash-0, groupId=logstash-staging-dc01] Subscribed to topic(s): weblog
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.101 [kafka-input-worker-logstash-0] ConsumerCoordinator - [Consumer clientId=logstash-0, groupId=logstash-staging-dc01] Resetting generation and member id due to: consumer pro-actively leaving the group
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.102 [kafka-input-worker-logstash-0] ConsumerCoordinator - [Consumer clientId=logstash-0, groupId=logstash-staging-dc01] Request joining group due to: consumer pro-actively leaving the group
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.103 [kafka-input-worker-logstash-0] Metrics - Metrics scheduler closed
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.104 [kafka-input-worker-logstash-0] Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.104 [kafka-input-worker-logstash-0] Metrics - Metrics reporters closed
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.106 [kafka-input-worker-logstash-0] ConsumerCoordinator - [Consumer clientId=logstash-0, groupId=logstash-syslog-staging-dc01] Resetting generation and member id due to: consumer pro-actively leaving the group
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.107 [kafka-input-worker-logstash-0] ConsumerCoordinator - [Consumer clientId=logstash-0, groupId=logstash-syslog-staging-dc01] Request joining group due to: consumer pro-actively leaving the group
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.108 [kafka-input-worker-logstash-0] Metrics - Metrics scheduler closed
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.109 [kafka-input-worker-logstash-0] Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.109 [kafka-input-worker-logstash-0] Metrics - Metrics reporters closed
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.121 [kafka-input-worker-logstash-0] AppInfoParser - App info kafka.consumer for logstash-0 unregistered
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.124 [kafka-input-worker-logstash-0] AppInfoParser - App info kafka.consumer for logstash-0 unregistered
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.246 [[main]>worker1] elasticsearch - Aborting the batch due to shutdown request while waiting for connections to become live
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.287 [[main]>worker1] WorkerLoop - Received signal to abort processing current batch. Terminating pipeline worker [main]>worker1
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.420 [[main]>worker0] elasticsearch - Aborting the batch due to shutdown request while waiting for connections to become live
Jan 18 11:15:45 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:45.422 [[main]>worker0] WorkerLoop - Received signal to abort processing current batch. Terminating pipeline worker [main]>worker0
Jan 18 11:15:46 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:46.178 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
Jan 18 11:15:46 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:46.612 [Converge PipelineAction::Delete<main>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:main}
Jan 18 11:15:46 logkafkadist01 logstash[4020950]: [INFO ] 2024-01-18 11:15:46.675 [LogStash::Runner] runner - Logstash shut down.
Jan 18 11:15:46 logkafkadist01 systemd[1]: logstash-staging.service: Succeeded.
Jan 18 11:15:46 logkafkadist01 systemd[1]: logstash-staging.service: Consumed 1min 39.475s CPU time.
rsommer commented 9 months ago

Despite the error being triggered by the AppInfo mbean, setting different client_ids for the manual configured kafka input instances fixed this.