search

logstash-plugins / logstash-filter-aggregate

The aim of this filter is to aggregate informations available among several events (typically log lines) belonging to a same task, and finally push aggregated information into final task event.
Apache License 2.0
43 stars 40 forks source link

testing aggregates => LogStash::ConfigurationError: Aggregate plugin: more than one filter which defines timeout options. But only defining once. #119

Open apple-corps opened 2 years ago

apple-corps commented 2 years ago

I'm trying to test my logstash configurations that use the aggregate plugin. I have not had any issues running them locally against the binary as I write them. But now I want to test the behavior expectations of the aggregations. Then I found a blog post and docker container that utilize rspec to test the filters. However I was trying to write a test to check the value of eventId and this ConfigurationError pops up. I'm surprised because it seems to me I'm only defining the timeout option in one aggregate block. But perhaps I'm doing something terribly wrong.

filter.conf

 fillter{
  dissect {
    mapping => {
      "message" => "%{emTimestamp}|%{}%{}| [%{eventId}] %{messageType}: %{msg}"
    }
  }

  if [messageType] == "GET Request" {
    aggregate {
      task_id => "%{eventId}"
      code => "
        map['eventId'] ||= event.get('eventId')
        map['emTimestamp'] ||= event.get('emTimestamp')
        map['actionType'] ||= 'aggregation'
        map['request'] = {'clientSource' => event.get('clientSource'), 'method' => event.get('method'), 'urlPath'=> event.get('urlPath')}"
    }
  }

  if [messageType] == "Response" {
    mutate { gsub => [ "msg","[\\]",""] }
    kv {
      source => "msg"
      value_split => "="
      field_split => "&?&"
      trim_key => "\s"
      trim_value => "\r"
      remove_field => ["msg", "message"]
    }

  json {
    source => ['body']
    target => "parsed"
  }

  ruby {
    code => "mapped = event.get('parsed').map {
      |h|
        {
          'obTimestamp' => h['item']['assets']['timestamp'],
          'fieldId'     => h['item']['assets']['reference']['fieldId'],
          'eventId'     => h['item']['assets']['reference']['eventId'],
          'userId'     => h['item']['assets']['reference']['userId'],
          'startTs'     => h['item']['attributes']['startTs'],
          'endTs'     => h['item']['attributes']['endTs']
        }
    }; event.set('mapped',mapped)"
  }

  aggregate {
    task_id => "%{eventId}"
    code => "
      map['eventId'] ||= event.get('eventId')
      map['emTimestamp'] ||= event.get('emTimestamp')
      map['actionType'] ||= 'aggregration'
      map['response'] = {'body' => event.get('mapped')}"
    }
  }

  if [messageType] == "ACK Response" {
    kv {
      source => "msg"
      value_split => "="
      field_split => "&,&"
      trim_key => "\s"
      trim_value => "\r"
      remove_field => ["msg"]
    }

    aggregate {
      task_id => "{eventId}"
      code => "
        map['eventId'] ||= event.get('eventId')
        map['emTimestamp'] ||= event.get('emTimestamp')
        map['actionType'] ||= 'aggregration'
      "
      push_map_as_event_on_timeout => true
      timeout_task_id_field => "eventId"
      timeout => 60
    }
  }
}

test_spec.rb

require "logstash/devutils/rspec/spec_helper"

# Load the configuration file
@@configuration = String.new
@@configuration << File.read("conf/filter.conf")

describe "Test ACK response type" do

  config(@@configuration)

  # Inject input event/message into the pipeline
  message = File.read("/opt/logstash/logs/tc.log")
  sample("message" => message, "type" => "") do
    # Check the ouput event/message properties
    insist { subject.get("transactionId")  == 104}
    reject { subject.get("tags").include?("_grokparsefailure") }
    reject { subject.get("tags").include?("_dateparsefailure") }
  end
end

error message and test failure

Failures:

  1) Test ACK response type "{"message":"10/23/2021 02:23:23 PM|DEBUG|HttpUtils ..." when processed
     Failure/Error: insist { subject.get("eventId")==104 } 

     LogStash::ConfigurationError:
       Aggregate plugin: For task_id pattern '%{eventId}', there are more than one filter which defines timeout options. All timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : timeout, inactivity_timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_timestamp_field, timeout_task_id_field, timeout_tags
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-filter-aggregate-2.9.1/lib/logstash/filters/aggregate.rb:103:in `block in register'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-filter-aggregate-2.9.1/lib/logstash/filters/aggregate.rb:97:in `register'
     # org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'
     # ./logstash-core/lib/logstash/pipeline.rb:288:in `register_plugin'
     # ./logstash-core/lib/logstash/pipeline.rb:299:in `block in register_plugins'
     # ./logstash-core/lib/logstash/pipeline.rb:299:in `register_plugins'
     # ./logstash-core/lib/logstash/pipeline.rb:660:in `maybe_setup_out_plugins'
     # ./logstash-core/lib/logstash/pipeline.rb:549:in `filter'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-1.3.6-java/lib/logstash/devutils/rspec/logstash_helpers.rb:54:in `block in results'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-1.3.6-java/lib/logstash/devutils/rspec/logstash_helpers.rb:52:in `block in results'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-1.3.6-java/lib/logstash/devutils/rspec/logstash_helpers.rb:68:in `block in subject'
     # ./rspec-tests/test_spec.rb:15:in `block in <main>'
     # ./vendor/bundle/jruby/2.5.0/gems/insist-1.0.0/lib/insist.rb:47:in `value'
     # ./vendor/bundle/jruby/2.5.0/gems/insist-1.0.0/lib/insist/comparators.rb:13:in `=='
     # ./rspec-tests/test_spec.rb:15:in `block in <main>'
     # ./vendor/bundle/jruby/2.5.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in <main>'
     # ./lib/bootstrap/rspec.rb:31:in `<main>'

Finished in 0.76796 seconds (files took 1.54 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./rspec-tests/test_spec.rb:13 # Test ACK response type "{"message":"10/23/2021 02:23:23 PM|DEBUG|HttpUtils ..." when processed

tc.log

10/23/2021 02:23:23 PM|DEBUG|HttpUtils                              | [104] Response: body =[{},"assets":{},{}],},"part":0}]

To reproduce launch this docker container and follow instructions to volume mount the filter, spec, and tc.log so the text executes. https://github.com/iteratec/logstash-rspec

docker command, populate paths.

    docker run --rm --name logstash-rspecs \
        -v /<path to filter dir>:/opt/logstash/filters-under-test \
        -v /<path to spec dir>:/opt/logstash/rspec-tests  \
        -v /<path to log dir>:/opt/logstash/logs \
        iteratec/logstash-rspec:7.3.0
apple-corps commented 2 years ago

I set workers to 1 in a configuration and enabled debugging and still hung up on this. Not sure about the scripting support message. Still failing due to the aggregate ConfigurationError.

Using system java: /usr/bin/java
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2021-11-29 07:51:02,139 main DEBUG Apache Log4j Core 2.14.0 initializing configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@7d1cfe97
2021-11-29 07:51:02,145 main INFO Cannot initialize scripting support because this JRE does not support it. java.lang.NoClassDefFoundError: javax/script/ScriptEngineFactory
    at java.base/java.lang.ClassLoader.findBootstrapClass(Native Method)
    at java.base/java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1258)
    at java.base/java.lang.System$2.findBootstrapClassOrNull(System.java:2134)
    at java.base/jdk.internal.loader.ClassLoaders$BootClassLoader.loadClassOrNull(ClassLoaders.java:118)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:616)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:640)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:616)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:576)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1210)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
    at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
    at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
    at java.scripting/javax.script.ScriptEngineManager.initEngines(ScriptEngineManager.java:123)
    at java.scripting/javax.script.ScriptEngineManager.init(ScriptEngineManager.java:87)
    at java.scripting/javax.script.ScriptEngineManager.<init>(ScriptEngineManager.java:62)
    at org.apache.logging.log4j.core.script.ScriptManager.<init>(ScriptManager.java:70)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:219)
    at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:68)
    at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:40)
    at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:559)
    at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:483)
    at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:712)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:267)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:597)
    at org.logstash.secret.store.SecretStoreFactory.<clinit>(SecretStoreFactory.java:75)
    at org.logstash.secret.store.SecretStoreExt.<clinit>(SecretStoreExt.java:32)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at org.jruby.javasupport.JavaSupportImpl.loadJavaClass(JavaSupportImpl.java:157)
    at org.jruby.javasupport.JavaSupportImpl.loadJavaClassVerbose(JavaSupportImpl.java:166)
    at org.jruby.javasupport.JavaClass.forNameVerbose(JavaClass.java:273)
    at org.jruby.javasupport.JavaClass.for_name(JavaClass.java:292)
    at org.jruby.javasupport.JavaClass.for_name(JavaClass.java:288)
    at org.jruby.javasupport.Java.get_proxy_class(Java.java:396)
    at org.jruby.javasupport.JavaUtilities.get_proxy_class(JavaUtilities.java:39)
    at org.jruby.javasupport.JavaUtilities$INVOKER$s$1$0$get_proxy_class.call(JavaUtilities$INVOKER$s$1$0$get_proxy_class.gen)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:172)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:160)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)
    at org.jruby.runtime.InterpretedIRBlockBody.yieldDirect(InterpretedIRBlockBody.java:107)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
    at org.jruby.runtime.Block.yield(Block.java:184)
    at org.jruby.RubyArray.collect(RubyArray.java:2563)
    at org.jruby.RubyArray.map19(RubyArray.java:2577)
    at org.jruby.RubyArray$INVOKER$i$0$0$map19.call(RubyArray$INVOKER$i$0$0$map19.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:93)
    at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:546)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:160)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
    at org.jruby.internal.runtime.methods.InterpretedIRMethod.INTERPRET_METHOD(InterpretedIRMethod.java:155)
    at org.jruby.internal.runtime.methods.InterpretedIRMethod.call(InterpretedIRMethod.java:146)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:375)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:174)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:96)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:81)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:30)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:1218)
    at org.jruby.Ruby.loadFile(Ruby.java:2785)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:234)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:887)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:535)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:402)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:981)
    at org.jruby.RubyKernel.require(RubyKernel.java:974)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require.call(RubyKernel$INVOKER$s$1$0$require.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:417)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:97)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:375)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:174)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:96)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:81)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:30)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:1218)
    at org.jruby.Ruby.loadFile(Ruby.java:2785)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:234)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:887)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:535)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:402)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:981)
    at org.jruby.RubyKernel.require(RubyKernel.java:974)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require.call(RubyKernel$INVOKER$s$1$0$require.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:417)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:97)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:375)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:174)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:96)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:81)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:30)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:1218)
    at org.jruby.Ruby.loadFile(Ruby.java:2785)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:234)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:887)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:535)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:402)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:981)
    at org.jruby.RubyKernel.require(RubyKernel.java:974)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require.call(RubyKernel$INVOKER$s$1$0$require.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:417)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:97)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:375)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:174)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:96)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:81)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:30)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:1240)
    at org.jruby.Ruby.runInterpreter(Ruby.java:1244)
    at org.jruby.Ruby.runNormally(Ruby.java:1133)
    at org.jruby.Ruby.runNormally(Ruby.java:1146)
    at org.jruby.Ruby.runFromMain(Ruby.java:958)
    at org.jruby.Main.doRunFromMain(Main.java:400)
    at org.jruby.Main.internalRun(Main.java:292)
    at org.jruby.Main.run(Main.java:234)
    at org.jruby.Main.main(Main.java:206)

2021-11-29 07:51:02,147 main DEBUG PluginManager 'Core' found 125 plugins
2021-11-29 07:51:02,148 main DEBUG PluginManager 'Level' found 0 plugins
2021-11-29 07:51:02,149 main DEBUG PluginManager 'Lookup' found 16 plugins
2021-11-29 07:51:02,151 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2021-11-29 07:51:02,157 main DEBUG PluginManager 'TypeConverter' found 26 plugins
2021-11-29 07:51:02,165 main DEBUG createAppenderRef(ref="console", level="null", Filter=null)
2021-11-29 07:51:02,166 main DEBUG Building Plugin[name=root, class=org.apache.logging.log4j.core.config.LoggerConfig$RootLogger].
2021-11-29 07:51:02,167 main DEBUG createLogger(additivity="null", level="DEBUG", includeLocation="null", ={console}, ={}, Configuration(LogstashRspecConfig), Filter=null)
2021-11-29 07:51:02,169 main DEBUG Building Plugin[name=loggers, class=org.apache.logging.log4j.core.config.LoggersPlugin].
2021-11-29 07:51:02,170 main DEBUG createLoggers(={root})
2021-11-29 07:51:02,170 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
2021-11-29 07:51:02,173 main DEBUG PatternLayout$Builder(pattern="[%d{ISO8601}][%-5p][%-25c] %m%n", PatternSelector=null, Configuration(LogstashRspecConfig), Replace=null, charset="null", alwaysWriteExceptions="null", disableAnsi="null", noConsoleNoAnsi="null", header="null", footer="null")
2021-11-29 07:51:02,173 main DEBUG PluginManager 'Converter' found 46 plugins
2021-11-29 07:51:02,183 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.ConsoleAppender].
2021-11-29 07:51:02,188 main DEBUG ConsoleAppender$Builder(target="null", follow="null", direct="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", PatternLayout([%d{ISO8601}][%-5p][%-25c] %m%n), name="console", Configuration(LogstashRspecConfig), Filter=null, ={})
2021-11-29 07:51:02,190 main DEBUG Starting OutputStreamManager SYSTEM_OUT.false.false
2021-11-29 07:51:02,190 main DEBUG Building Plugin[name=appenders, class=org.apache.logging.log4j.core.config.AppendersPlugin].
2021-11-29 07:51:02,190 main DEBUG createAppenders(={console})
2021-11-29 07:51:02,191 main DEBUG Configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@7d1cfe97 initialized
2021-11-29 07:51:02,196 main DEBUG Starting configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@7d1cfe97
2021-11-29 07:51:02,197 main DEBUG Started configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@7d1cfe97 OK.
2021-11-29 07:51:02,198 main DEBUG Shutting down OutputStreamManager SYSTEM_OUT.false.false-1
2021-11-29 07:51:02,199 main DEBUG OutputStream closed
2021-11-29 07:51:02,199 main DEBUG Shut down OutputStreamManager SYSTEM_OUT.false.false-1, all resources released: true
2021-11-29 07:51:02,200 main DEBUG Appender DefaultConsole-1 stopped with status true
2021-11-29 07:51:02,200 main DEBUG Stopped org.apache.logging.log4j.core.config.DefaultConfiguration@3b11deb6 OK
2021-11-29 07:51:02,273 main DEBUG Registering MBean org.apache.logging.log4j2:type=2530c12
2021-11-29 07:51:02,276 main DEBUG Registering MBean org.apache.logging.log4j2:type=2530c12,component=StatusLogger
2021-11-29 07:51:02,277 main DEBUG Registering MBean org.apache.logging.log4j2:type=2530c12,component=ContextSelector
2021-11-29 07:51:02,278 main DEBUG Registering MBean org.apache.logging.log4j2:type=2530c12,component=Loggers,name=
2021-11-29 07:51:02,279 main DEBUG Registering MBean org.apache.logging.log4j2:type=2530c12,component=Appenders,name=console
2021-11-29 07:51:02,281 main DEBUG org.apache.logging.log4j.core.util.SystemClock supports precise timestamps.
2021-11-29 07:51:02,281 main DEBUG Reconfiguration complete for context[name=2530c12] at URI /src/logstash/log4j2.properties (org.apache.logging.log4j.core.LoggerContext@2433bcd4) with optional ClassLoader: null
2021-11-29 07:51:02,282 main DEBUG Shutdown hook enabled. Registering a new one.
2021-11-29 07:51:02,283 main DEBUG LoggerContext[name=2530c12, org.apache.logging.log4j.core.LoggerContext@2433bcd4] started OK.
Sending Logstash logs to null which is now configured via log4j2.properties
2021-11-29 07:51:04,476 main DEBUG Reconfiguration started for context[name=2530c12] at URI file:////src/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-devutils-1.3.6-java/lib/logstash/devutils/rspec/log4j2.properties (org.apache.logging.log4j.core.LoggerContext@2433bcd4) with optional ClassLoader: null
2021-11-29 07:51:04,476 main DEBUG Using configurationFactory org.apache.logging.log4j.core.config.ConfigurationFactory$Factory@3dc70aa0
2021-11-29 07:51:04,477 main DEBUG Not in a ServletContext environment, thus not loading WebLookup plugin.
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}

Randomized with seed 6451
F

Failures:

  1) Test single log line with 3 trips in response body "09/28/2021 02:45:18 PM|INFO |poll                  ..." when processed
     Failure/Error: expect (subject.get("topic").to eq("2304053812") )

     LogStash::ConfigurationError:
       Aggregate plugin: For task_id pattern '%{transactionId}', there are more than one filter which defines timeout options. All timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : timeout, inactivity_timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_timestamp_field, timeout_task_id_field, timeout_tags
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-filter-aggregate-2.9.1/lib/logstash/filters/aggregate.rb:103:in `block in register'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-filter-aggregate-2.9.1/lib/logstash/filters/aggregate.rb:97:in `register'
     # org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'
     # ./logstash-core/lib/logstash/pipeline.rb:288:in `register_plugin'
     # ./logstash-core/lib/logstash/pipeline.rb:299:in `block in register_plugins'
     # ./logstash-core/lib/logstash/pipeline.rb:299:in `register_plugins'
     # ./logstash-core/lib/logstash/pipeline.rb:674:in `maybe_setup_out_plugins'
     # ./logstash-core/lib/logstash/pipeline.rb:550:in `filter'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-1.3.6-java/lib/logstash/devutils/rspec/logstash_helpers.rb:54:in `block in results'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-1.3.6-java/lib/logstash/devutils/rspec/logstash_helpers.rb:52:in `block in results'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-1.3.6-java/lib/logstash/devutils/rspec/logstash_helpers.rb:68:in `block in subject'
     # /work/spec/02_filter_spec.rb:124:in `block in <main>'
     # ./vendor/bundle/jruby/2.5.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in <main>'
     # ./lib/bootstrap/rspec.rb:31:in `<main>'

Finished in 0.58592 seconds (files took 1.48 seconds to load)
1 example, 1 failure

Failed examples:

rspec /work/spec/02_filter_spec.rb:118 # Test single log line with 3 trips in response body "09/28/2021 02:45:18 PM|INFO |poll                  ..." when processed

Randomized with seed 6451
apple-corps commented 2 years ago

Another set of reproduction steps given here: https://github.com/cameronkerrnz/logstash-plugin-dev/issues/5#issuecomment-982284743

apple-corps commented 2 years ago

Assuming 2021-11-29 07:51:02,145 main INFO Cannot initialize scripting support because this JRE does not support it. java.lang.NoClassDefFoundError: javax/script/ScriptEngineFactory

is unrelated to my issue.