search

logstash-plugins / logstash-input-azure_event_hubs

Logstash input for consuming events from Azure Event Hubs
Apache License 2.0
18 stars 28 forks source link

Logstash 7.10: Unable to configure plugins: (NoMethodError) undefined method `each_index' #64

Closed etiennetremel closed 2 years ago

etiennetremel commented 3 years ago

Logstash 7.10.0 fail loading the following configuration:

input {
  azure_event_hubs {
    event_hub_connections => ["${AZURE_EVENTHUB_CONNECTION_STRING}"]
    decorate_events => true
    consumer_group => "logstash"
    storage_connection => "${AZURE_STORAGE_CONNECTION_STRING}"
    storage_container => "logstash"
    tags => ["azure-functions"]
  }
}

It was working fine with Logstash 7.9.3 but not with 7.10.0.

I'm running Logstash in Docker using the docker.elastic.co/logstash/logstash-oss:7.10.0 image.

Logs:

[2020-11-27T09:21:19,361][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@consumer_group = "logstash"
[2020-11-27T09:21:19,363][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@storage_container = "logstash"
[2020-11-27T09:21:19,365][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@event_hub_connections = ["Endpoint=sb://<redacted>.servicebus.windows.net/;SharedAccessKeyName=logstash;SharedAccessKey=<redacted>;EntityPath=<redacted>"]
[2020-11-27T09:21:19,369][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@id = "<redacted>"
[2020-11-27T09:21:19,375][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@storage_connection = <password>
[2020-11-27T09:21:19,376][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@decorate_events = true
[2020-11-27T09:21:19,378][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@tags = ["azure-functions"]
[2020-11-27T09:21:19,379][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@event_hubs = ["dummy"]
[2020-11-27T09:21:19,380][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@enable_metric = true
[2020-11-27T09:21:19,395][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@codec = <LogStash::Codecs::Plain id=>"plain_<redacted>", enable_metric=>true, charset=>"UTF-8">
[2020-11-27T09:21:19,396][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@add_field = {}
[2020-11-27T09:21:19,397][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@config_mode = "basic"
[2020-11-27T09:21:19,398][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@threads = 16
[2020-11-27T09:21:19,399][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@max_batch_size = 125
[2020-11-27T09:21:19,400][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@prefetch_count = 300
[2020-11-27T09:21:19,400][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@receive_timeout = 60
[2020-11-27T09:21:19,402][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@initial_position = "beginning"
[2020-11-27T09:21:19,403][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@initial_position_look_back = 86400
[2020-11-27T09:21:19,404][DEBUG][logstash.inputs.azureeventhubs] config LogStash::Inputs::AzureEventHubs/@checkpoint_interval = 5
[2020-11-27T09:21:19,430][FATAL][logstash.runner          ] The given configuration is invalid. Reason: Unable to configure plugins: (NoMethodError) undefined method `each_index' for <Java::JavaUtil::ArrayList:-980064772 ["azure-functions"]>:Java::JavaUtil::ArrayList
[2020-11-27T09:21:19,439][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Additional informations (Gem.lock):

    logstash-core (7.10.0-java)
      chronic_duration (~> 0.10)
      clamp (~> 0.6)
      concurrent-ruby (~> 1)
      elasticsearch (~> 5)
      filesize (~> 0.2)
      gems (~> 1)
      i18n (~> 1)
      jrjackson (= 0.4.12)
      jruby-openssl (= 0.10.4)
      manticore (~> 0.6)
      minitar (~> 0.8)
      mustermann (~> 1.0.3)
      pry (~> 0.12)
      puma (~> 4)
      rack (~> 2)
      rubyzip (~> 1)
      sinatra (~> 2)
      stud (~> 0.0.19)
      thread_safe (~> 0.3.6)
      treetop (~> 1)

[...]

    logstash-input-azure_event_hubs (1.2.2)
      logstash-codec-json
      logstash-codec-plain
      logstash-core-plugin-api (~> 2.0)
      stud (>= 0.0.22)

Any clue?

cskowronnek commented 3 years ago

Same here with Logstash 7.10.1 installed as package on Ubuntu.

jdepalma commented 3 years ago

Getting the same thing after upgrading from logstash 7.8.1 to 7.11.0

I downgraded back down to 7.8.1, updated the plugins, and the plugin resumed normal function. Still works on 7.8.1, not on 7.11

KrishnanandSingh commented 3 years ago

I think this is a generic problem with arrays in this plugin. Below example also does not work. I was going to raise a new issue but this seems same. Logstash version: 7.12.0

input {
  azure_event_hubs {
    event_hub_connections => ["${AZURE_EVENTHUB_CONNECTION_STRING}"]
    decorate_events => true
    consumer_group => "logstash"
    storage_connection => "${AZURE_STORAGE_CONNECTION_STRING}"
    storage_container => "logstash"
    add_field => {
        "field_name" => ["value"]
    }
  }
}
cskowronnek commented 3 years ago

any news? we're still stuck at LS 7.9.3

Mehdi-Mameri commented 2 years ago

Hi,

Any news on it, with the 7.15 version of Logstash it always failed here logs of service logstash (7.15)

[Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-11-15 09:59:39.525 [main] runner - Starting Logstash {"logstash.version"=>"7.15.1", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.12+7 on 11.0.12+7 +indy +jit [linux-x86_64]"}
[INFO ] 2021-11-15 09:59:39.551 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2021-11-15 09:59:39.561 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2021-11-15 09:59:39.896 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-11-15 09:59:39.919 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"5fe5539e-0186-44af-815b-49879d15703c", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2021-11-15 09:59:41.303 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-11-15 09:59:41.823 [Converge PipelineAction::Create<main>] Reflections - Reflections took 98 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2021-11-15 09:59:42.674 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[ERROR] 2021-11-15 09:59:42.733 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (NoMethodError) undefined method `each_index' for #<Java::JavaUtil::ArrayList:0x76645a38>", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:391)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:135
             add at org/logstash/execution/ConvergeResultExt.java:60
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:404
[ERROR] 2021-11-15 09:59:42.741 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[FATAL] 2021-11-15 09:59:42.752 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:404:in `block in converge_state'"]}
[FATAL] 2021-11-15 09:59:42.778 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
    at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]]
githubcdr commented 2 years ago

Can this be raised concerning the latest log4j issues?

kaisecheng commented 2 years ago

This issue is fixed in https://github.com/elastic/logstash/pull/13603 and released in Logstash v7.17.0