[discuss] default plugins dependencies smoke test proposal

[colinsurprenant]

Relates to #7627 - [meta] streamline and harmonize plugins with jars build and publish processes

This configuration contains all the default plugins

input {
  stdin { codec => collectd { typesdb => "./bin/logstash" }}
  stdin { codec => dots {}}
  stdin { codec => edn {}}
  stdin { codec => edn_lines {}}
  stdin { codec => fluent {}}
  stdin { codec => es_bulk {}}
  stdin { codec => graphite {}}
  stdin { codec => json {}}
  stdin { codec => json_lines {}}
  stdin { codec => line {}}
  stdin { codec => msgpack {}}
  stdin { codec => multiline { pattern => "foo" what => "next" }}
  stdin { codec => netflow {}}
  stdin { codec => plain {}}
  stdin { codec => rubydebug {}}

  heartbeat {}
  #dead_letter_queue {} # fails with missing jar
  couchdb_changes { db => "foo" }
  elasticsearch {}
  exec { command => "" interval => "1" }
  file { path => "foo" }
  ganglia {}
  gelf {}
  generator {}
  graphite { port => "666" }
  http {}
  http_poller { urls => { "foo" => "bar" }}
  imap { host => "foo" user => "bar" password => "baz" }
  irc { host => "foo" channels => "bar" }
  jdbc { jdbc_driver_class => "foo" jdbc_connection_string => "bar" jdbc_user => "baz" }
  log4j {}
  lumberjack { port => 666 ssl_certificate => "./bin/logstash" ssl_key => "./bin/logstash" }
  pipe { command => "foo" }
  rabbitmq { host => "foo" }
  redis { key => "foo" data_type => "list" }
  s3 { bucket => "foo" }
  snmptrap {}
  sqs { queue => "foo" }
  stdin {}
  syslog {}
  tcp { port => 666 }
  twitter { consumer_key => "foo" consumer_secret => "bar" oauth_token => "baz" oauth_token_secret => "biz" }
  udp { port => 666 }
  unix { path => "foo" }
  xmpp { user => "foo" password => "bar" }
  kafka {}
  beats { port => 666 }
}

filter {
  clone {}
  csv {}
  date { match => [ "foo", "M" ]}
  dissect {}
  dns {}
  drop {}
  fingerprint {}
  geoip { source => "foo" }
  grok {}
  json { source => "foo" }
  kv {}
  metrics {}
  mutate {}
  ruby { code => "" }
  sleep {}
  split {}
  syslog_pri {}
  throttle { key => "foo" }
  urldecode {}
  useragent { source => "foo" }
  uuid { target => "foo" }
  xml { source => "foo" }
}

output {
  xmpp { user => "foo" password => "bar" message => "baz" }
  cloudwatch {}
  csv { path => "foo" fields => "bar" }
  elasticsearch {}
  file { path => "foo" }
  graphite {}
  http { url => "foo" http_method => "post" }
  irc { host => "foo" channels => "bar" }
  kafka { topic_id => "foo" }
  nagios {}
  null {}
  pagerduty { service_key => "foo" }
  pipe { command => "foo" }
  rabbitmq { host => "foo" exchange => "bar" exchange_type => "direct" }
  redis {}
  s3 { bucket => "foo" }
  sns {}
  sqs { queue => "foo" }
  statsd {}
  stdout {}
  tcp { host => "foo" port => 666 }
  udp { host => "foo" port => 666 }
  webhdfs { host => "foo" user => "bar" path => "baz" }
}

and can be used with this command to test the loading/initialization and dependencies loading (including jar files) with

bin/logstash -t -f default-plugins.conf

If all plugins are successfully loaded, then logstash exits with a shell exit status of 0 and this output

Configuration OK
[2017-07-12T13:38:10,286][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Otherwise it will exit with and error log and a shell exit status of 1, for example by uncommenting thedead_letter_queue` input in the above config

[2017-07-12T13:37:13,765][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"input", :name=>"dead_letter_queue", :path=>"logstash/inputs/dead_letter_queue", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:226:in `namespace_lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:162:in `legacy_lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in `lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/pipeline.rb:125:in `plugin'", "(eval):72:in `<eval>'", "org/jruby/RubyKernel.java:1000:in `eval'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/pipeline.rb:82:in `initialize'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/runner.rb:320:in `block in execute'", "org/jruby/RubyArray.java:1734:in `each'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/runner.rb:319:in `execute'", "/Users/colin/dev/src/elasticsearch/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/runner.rb:219:in `run'", "/Users/colin/dev/src/elasticsearch/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/Users/colin/dev/src/elasticsearch/logstash/lib/bootstrap/environment.rb:71:in `<main>'"]}
[2017-07-12T13:37:13,769][FATAL][logstash.runner          ] The given configuration is invalid. Reason: Couldn't find any input plugin named 'dead_letter_queue'. Are you sure this is correct? Trying to load the dead_letter_queue input plugin resulted in this error: Problems loading the requested plugin named dead_letter_queue of type input. Error: NameError NameError

Proposal

• [ ] add a new field in logstash/rakelib/plugins-metadata.json named sample-config: "..." which will contain a string of a syntax-valid plugin config, basically the plugin configs that are required as above in the default plugins config file.
• [ ] create a task to generate the default plugins config file based on logstash/rakelib/plugins-metadata.json for the "default-plugins": true and using the sample-config: "..." field.
• [ ] add a package generation smoke test to execute bin/logstash -t -f default-plugins.conf and check for exit status

Note: the only tricky config of all default plugins is with the lumberjack input and collectd codec which actually verifies the existence of the configured files. We can decide how to deal with it, but for now I just tricked it with a file I know is part of the package :)

[andrewvc]

I'm wondering how this compares to having a new logstash command that runs list_of_plugin_names.each {|p_name| LogStash::Plugin.lookup.(p_name)} which was, I believe, proposed by @jsvd ? The nice thing about that is that we wouldn't need to maintain the sample-config stuff.

[colinsurprenant]

@andrewvc The difference is that this proposal is a real out-of-the-box test which calls logstash on a real config on a generated package. it is much more closer to what happens when a user spins logstash so a good way to validate the integrity of a package related to its actual bundled plugins.

[suyograo]

My 2c:

I personally like that we load these plugins like the user would (using bin/logstash) when using the mega config. As such this would make it a real integration test which is the spirit of the RATS test.

One concern is about the overhead of maintaining a config field in the plugins-metadata.json and keeping it in sync, but we're only using the bare minimum config to trigger the load, so this shouldn't be a big deal.

Thinking further, this kind of test can have a nice side effect. We can catch any unintentional backward incompatibility issues with the configs.

[andrewvc]

OK, I'm +1 on this approach.

[colinsurprenant]

One concern is about the overhead of maintaining a config field in the plugins-metadata.json and keeping it in sync, but we're only using the bare minimum config to trigger the load, so this shouldn't be a big deal.

Agree, also plugins required settings is not something we change often and it won't cause "silent" upstream problems since any mismatch will result in the test failing, in that respect it is safe. But yes, this new sample-config: "..." field might need to be updated from time to time if we change a plugin config required settings.

[jsvd]

This is a quick win, regardless of using .lookup or -t, with the added benefit the latter runs the initialize method of each plugin.

As long as we don't have plugins that instantiate resources that can create conflicts in their constructor (and they shouldn't, that's what register is for), it should be hassle free.

+1