logstash-plugins / logstash-input-azure_event_hubs

Logstash input for consuming events from Azure Event Hubs
Apache License 2.0
18 stars 28 forks source link

Azure EventHubs Input plugin fails to initialize on Logstash v8.14.x, works in 8.13.x #91

Open heironeous opened 2 months ago

heironeous commented 2 months ago

Logstash information:

Please include the following information:

  1. Logstash version: 8.14.1 & 8.14.0
  2. Logstash installation source: Docker
  3. How is Logstash being run: Docker - using docker-entrypoint
  4. How was the Logstash Plugin installed: Comes installed by default

JVM (e.g. java -version): Temurin-17.0.11+9 (comes preinstalled in docker container)

OS version (uname -a if on a Unix-like system): Running on ECS Fargate nodes

Description of the problem including expected versus actual behavior:

Azure EventHubs Input plugin fails during initialization on 8.14.0 and 8.14.1 It works perfectly fine on 8.13.x - we are currently running working version 8.13.4 The logs suggest an issue with the plugin, but I'm not sure if that's relevant

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

  1. Add this to dockerfile as input filter:
    input {
    azure_event_hubs {
    config_mode => "advanced"
    threads => "2"
    decorate_events => true
    type => "azure"
    event_hubs => [
      {
        "logs-vhub-ch-north-prod-01" => {
          event_hub_connection => "Endpoint=sb://fam-events-ch-north-prod-01.servicebus.windows.net/;SharedAccessKeyName=ACCESSKEYNAME;SharedAccessKey=ACCESSKEY=;EntityPath=logs-vhub-ch-north-prod-01"
          initial_position => "beginning"
          consumer_group => "logstash-production"
        }
      }
    ]
    }
    }
  2. Define any output plugin, such as stdout:
    output {
    stdout { codec => json_lines }
    }
  3. Run logstash

Provide logs (if relevant):

{
    "level": "ERROR",
    "loggerName": "logstash.inputs.azureeventhubs",
    "timeMillis": 1718892047629,
    "thread": "Ruby-0-Thread-30: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-azure_event_hubs-1.4.7/lib/logstash/inputs/azure_event_hubs.rb:381",
    "logEvent": {
        "message": "Event Hub failed during initialization.",
        "event_hub_name": "logs-vhub-ch-north-prod-01",
        "exception": {
            "stackTrace": [
                {
                    "class": "RUBY",
                    "method": "run",
                    "file": "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-azure_event_hubs-1.4.7/lib/logstash/inputs/azure_event_hubs.rb",
                    "line": 408
                }
            ],
            "exception": "undefined method `getHostContext' for #<Java::ComMicrosoftAzureEventprocessorhost::EventProcessorHost:0x4cf95ede>\nDid you mean?  get_host_name",
            "message": "(NoMethodError) undefined method `getHostContext' for #<Java::ComMicrosoftAzureEventprocessorhost::EventProcessorHost:0x4cf95ede>\nDid you mean?  get_host_name",
            "localizedMessage": "(NoMethodError) undefined method `getHostContext' for #<Java::ComMicrosoftAzureEventprocessorhost::EventProcessorHost:0x4cf95ede>\nDid you mean?  get_host_name"
        },
        "backtrace": [
            "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-azure_event_hubs-1.4.7/lib/logstash/inputs/azure_event_hubs.rb:408:in `block in run'"
        ]
    }
}
FelipeEmerimRandon commented 1 month ago

We're having the same issue, not fixed in Logstash 8.15. Checking the plugin version, logstash 8.13 uses version 1.4.5 and logstash 8.15 uses version 1.4.7, the CHANGELOG only shows documentation changes...

ofer-haim commented 3 days ago

I see the same problem

event_hub_connections => ['Endpoint=sb://<validURL>/;SharedAccessKeyName=<accessKeyName>;SharedAccessKey=<accessKey>;EntityPath=np-postgres-eventhub']

Worked in Logstash 8.13.4. Doesn't work in Logstash 8.14.x and 8.15.x. Seems related to the JRUBY upgrade to 9.4.7.0.

Getting the following error: [2024-09-11T12:59:45,487][FATAL][logstash.runner ] The given configuration is invalid. Reason: Unable to configure plugins: (ConfigurationError) Error parsing event hub string name for connection: '' please ensure that the connection string contains the EntityPath