Issue trying to get environment variable to be read in logstash conf

[wingZero21]

Hi,

Can anyone advise how we can get a logstash conf file to read from an system set environment variable.

I have tried several options but nothing seems to work.

I have added below what I have currently.

Details: filter {

environment { add_field_from_env => { "escluster" => "ESCLUSTER" } }

output {

stdout { } elasticsearch { cluster => "cluster1" protocol => "http" port => "9200" host => "%{escluster}" } }

Error: URI::InvalidURIError: bad URI(is not URI?): http://%{escluster}:9200 split at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:176 parse at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:210 parse at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:747 URI at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:994 call at org/jruby/RubyMethod.java:128 URI at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/utils.rb:246 url_prefix= at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/connection.rb:309 initialize at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/connection.rb:77 __build_connections at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/http/faraday.rb:42 map at org/jruby/RubyArray.java:2409 __build_connections at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/http/faraday.rb:34 initialize at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/base.rb:32 initialize at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/client.rb:90 new at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport.rb:26 build_client at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:62 client at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:15 initialize at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:58 register at /opt/logstash/lib/logstash/outputs/elasticsearch.rb:250 each at org/jruby/RubyArray.java:1613 outputworker at /opt/logstash/lib/logstash/pipeline.rb:220 start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152

[torrancew]

@wingZero21 I can confirm this behavior, based on the following:

# logstash.conf
input {
  stdin {}
}

filter {
  environment {
    add_field_from_env => {"escluster" => "ESCLUSTER"}
  }
}

output {
  stdout {
    codec => rubydebug
  }

  #elasticsearch {
  #  host => "%{escluster}"
  #  protocol => "http"
  #}
}

user@host:~/src/logstash $ export ESCLUSTER=foo

When run as-is:

user@host:~/src/logstash $ echo "Test" | ./bin/logstash -f logstash.conf
Using milestone 1 filter plugin 'environment'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
{
   "message" => "Test",
  "@version" => "1",
"@timestamp" => "2014-10-18T19:26:06.469Z",
      "host" => "host",
 "escluster" => "foo"
}

When run with the ES plugin enabled:

user@host:~/src/logstash $ echo "Test" | ./bin/logstash -f logstash.conf
Using milestone 1 filter plugin 'environment'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
URI::InvalidURIError: bad URI(is not URI?): http://%{escluster}:9200
                split at file:/home/tray/tmp/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:176
                parse at file:/home/tray/tmp/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:210
                parse at file:/home/tray/tmp/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:747
                  URI at file:/home/tray/tmp/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/uri/common.rb:994
                 call at org/jruby/RubyMethod.java:128
                  URI at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/utils.rb:246
          url_prefix= at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/connection.rb:309
           initialize at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/connection.rb:77
  __build_connections at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/http/faraday.rb:42
                  map at org/jruby/RubyArray.java:2409
  __build_connections at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/http/faraday.rb:34
           initialize at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/base.rb:32
           initialize at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/client.rb:90
                  new at /home/tray/tmp/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport.rb:26
         build_client at /home/tray/tmp/logstash-1.4.2/lib/logstash/outputs/elasticsearch/protocol.rb:62
               client at /home/tray/tmp/logstash-1.4.2/lib/logstash/outputs/elasticsearch/protocol.rb:15
           initialize at /home/tray/tmp/logstash-1.4.2/lib/logstash/outputs/elasticsearch/protocol.rb:58
             register at /home/tray/tmp/logstash-1.4.2/lib/logstash/outputs/elasticsearch.rb:250
                 each at org/jruby/RubyArray.java:1613
         outputworker at /home/tray/tmp/logstash-1.4.2/lib/logstash/pipeline.rb:220
        start_outputs at /home/tray/tmp/logstash-1.4.2/lib/logstash/pipeline.rb:152

Digging into the ES output, my suspicion is confirmed: Connections are established at plugin registration time (during initialization, as they almost certainly should be), but field interpolation (like "%{escluster}") is an event-processing-time operation. As such, "host" isn't really eligible for this behavior.

I'm not sure your exact use case, but we can probably help you figure out a workable substitute for this behavior.

@jordansissel This type of thing has come up a few times, and I wonder if we can find a way to mark fields that are interpolation-eligible as such in code, and ultimately plugin docs?

[jordansissel]

I can confirm:

1. The elasticsearch output 'host' setting does not support the %{foo} syntax.
2. It is intentional that this 'host' setting behaves this way. Until now, nobody's asked, that I can recall, about such af eature.
3. We should document this more clearly.

Recommendations -

If you are really just aiming to use an environment variable to populate the 'host' setting, you can use m4 (or any templating system) to achieve this prior to launching Logstash:

% cat /tmp/logstash.conf
output {
  elasticsearch {
    host => "SERVER"
  }
}

% server="es1.example.com"

% m4 -DSERVER="$server" < /tmp/logstash.conf
output {
  elasticsearch {
    host => "es1.example.com"
  }
}

So, you can template this by having your "start Logstash" workflow include a templating action that respects your desired template inputs (environment variables, etc)

[wingZero21]

Hi,

Yeah we have potential ways around this. The reasoning for wanting to use the Environment variable is that were going to be using RPM's for the configuration files and we have 2 DC's which are going to have a separate cluster for each DC. If we cant use the Environment variable its likely we will need to have 2 sets of the same configuration file one for DC1 and another for DC2 as the cluster names are unique for each DC.

Like has been mentioned this can be done either via separate RPM or configuration management however we were looking at using the environment variables to separate this out potentially if possible but thanks for the clarification as we were going a bit mad trying to understand why this was not working initially :D!