logstash-plugins / logstash-input-azure_event_hubs

Logstash input for consuming events from Azure Event Hubs
Apache License 2.0
18 stars 28 forks source link

When using the advanced configuration event_hub_connection and storage_connection are not read from the keystore #48

Open luizgpsantos opened 4 years ago

luizgpsantos commented 4 years ago

Here are the steps to reproduce this issue:

  1. Create the event_hub_connection setting in the logstash keystore:
$ echo "Endpoint=sb://REDACTED;SharedAccessKeyName=logstash;SharedAccessKey=REDACTED;EntityPath=REDACTED" | ./bin/logstash-keystore add event_hub_connection --stdin --force
  1. Create the pipeline:
input {
   azure_event_hubs {
      config_mode => "advanced"
      codec => json
      decorate_events => true
      event_hubs => [
        {"normal-priority" => {
            event_hub_connection => "${event_hub_connection}"
            consumer_group => "logstash"
            storage_connection => "DefaultEndpointsProtocol=https;AccountName=REDACTED;AccountKey=REDACTED;EndpointSuffix=core.windows.net"
            }
        }
      ]
  }
}

output { stdout {} }
  1. Run logstash
$ ./bin/logstash -f event_hub_keystore_test.conf
[2020-01-31T14:23:32,345][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.5.0"}
...
[2020-01-31T14:23:34,805][INFO ][logstash.inputs.azureeventhubs][main] Event Hub normal-priority is initializing...
[2020-01-31T14:23:34,841][ERROR][logstash.inputs.azureeventhubs][main] Event Hub failed during initialization. {:event_hub_name=>"REDACTED", :exception=>com.microsoft.azure.eventhubs.IllegalConnectionStringFormatException: Connection String cannot be parsed., :backtrace=>["com.microsoft.azure.eventhubs.ConnectionStringBuilder.parseConnectionString(com/microsoft/azure/eventhubs/ConnectionStringBuilder.java:343)", "com.microsoft.azure.eventhubs.ConnectionStringBuilder.<init>(com/microsoft/azure/eventhubs/ConnectionStringBuilder.java:103)", "com.microsoft.azure.eventprocessorhost.EventProcessorHost.<init>(com/microsoft/azure/eventprocessorhost/EventProcessorHost.java:232)", "com.microsoft.azure.eventprocessorhost.EventProcessorHost.<init>(com/microsoft/azure/eventprocessorhost/EventProcessorHost.java:167)", "com.microsoft.azure.eventprocessorhost.EventProcessorHost.<init>(com/microsoft/azure/eventprocessorhost/EventProcessorHost.java:152)", "com.microsoft.azure.eventprocessorhost.EventProcessorHost.<init>(com/microsoft/azure/eventprocessorhost/EventProcessorHost.java:98)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:490)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:247)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "REDACTED.logstash_minus_7_dot_5_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_azure_event_hubs_minus_1_dot_1_dot_2.lib.logstash.inputs.azure_event_hubs.run(/REDACTED/logstash-7.5.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.1.2/lib/logstash/inputs/azure_event_hubs.rb:386)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]}
[2020-01-31T14:23:34,850][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-01-31T14:23:35,059][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-01-31T14:23:36,488][INFO ][logstash.runner          ] Logstash shut down.