search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
75 stars 3.5k forks source link

Config environment variable not being read from /etc/environment #9007

Closed nodesocket closed 6 years ago

nodesocket commented 6 years ago

I am trying to use an environment variable in my Logstash configuration:

elasticsearch { hosts => ["${ELASTICSEARCH_ENDPOINT}"] }

But I am getting the error:

Cannot evaluate `${ELASTICSEARCH_ENDPOINT}`. Environment variable `ELASTICSEARCH_ENDPOINT` is not set and there is no default value given.

However when I ssh into the instance, the environment variable is indeed set:

echo $ELASTICSEARCH_ENDPOINT
search-myorg-elasticsearch-dev-gvxi25arm6xjn7y4.us-west-2.es.amazonaws.com

It is being set in /etc/environment. Is setting environment variables here supported? Running Ubuntu 16.04 and ZSH shell.

jakelandis commented 6 years ago

@nodesocket - are you running Logstash via the service ?

If so, the service runs as the logstash user which is a no shell user, and thus no shell environment variables (like /etc/environment). To set environment variables for the logstash service you should be able set it like this:

echo "ELASTICSEARCH_ENDPOINT=search-myorg-elasticsearch-dev-gvxi25arm6xjn7y4.us-west-2.es.amazonaws.com" | sudo tee --append /etc/sysconfig/logstash

If this is not from a deb/rpm package, does it work from the bash shell ?

jordansissel commented 6 years ago

/etc/environment

I don't think we promise that this file is used by Logstash, do we?

nodesocket commented 6 years ago

@jakelandis thank you very much for the reply and help. /etc/sysconfig/logstash does not exist, in fact the directory /etc/sysconfig does not exist. I am running Ubuntu.

I took a look at the systemd script in /etc/systemd/system/logstash.service and found these two lines:

EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash

💥 /etc/default/logstash is what I want. So the full command is:

echo "ELASTICSEARCH_ENDPOINT=search-myorg-elasticsearch-dev-gvxi25arm6xjn7y4.us-west-2.es.amazonaws.com" | sudo tee -a /etc/default/logstash > /dev/null

Thanks 💯 !

jordansissel commented 6 years ago

@nodesocket it's possible we need to improve the documentation (or logging? or both?) of how to configure Logstash as a service. Thanks for filing this issue :)

bhaveshf-cuelogic commented 5 years ago

I am also facing similar issue when running logstash via docker. I am passing environment variable using 

docker run -e MY_VAR=value....

which also is further available in the container. But the same environment variable's value is not picked up in logstash's .conf file where I am reading it using ${MY_VAR}. There it's taking ${MY_VAR}'s default value which is specified in keystore.

jordansissel commented 5 years ago

@bhaveshf-cuelogic if I try this, it works for me:

% docker run -e 'FOO=bar' docker.elastic.co/logstash/logstash:7.4.0 -e 'input { generator { count => 1 message => "${FOO}" } }'
{
       "message" => "bar",
      "@version" => "1",
      "sequence" => 0,
    "@timestamp" => 2019-10-10T17:28:03.663Z,
          "host" => "9dff431547ec"
}
bhaveshf-cuelogic commented 5 years ago

@bhaveshf-cuelogic if I try this, it works for me:

% docker run -e 'FOO=bar' docker.elastic.co/logstash/logstash:7.4.0 -e 'input { generator { count => 1 message => "${FOO}" } }'
{
       "message" => "bar",
      "@version" => "1",
      "sequence" => 0,
    "@timestamp" => 2019-10-10T17:28:03.663Z,
          "host" => "9dff431547ec"
}

@jordansissel : Yes, this works. Don't we need to store FOO in logstash keystore if we want to use it in config ?

This config does not work for me if I have stored it in .conf file.

docker build -t test-build:v1

followed by

docker run -p 5044:5044 -e KAFKA_TOPIC_ID=test test-build:v1

After I post some message on port 5044, I get output as:

{
  "@version" => "1",
  "kafkatopic" => "raw-logs121",
  "message" => "test hi",
  "type" => "tcp",
  "@timestamp" => 2019-10-10T19:33:07.305Z,
  "port" => 53544,
  "host" => "gateway"
}

Note that kafkatopic contains default value configured in keystore but not what is passed via -e option

jordansissel commented 5 years ago

Looks like this is intentional, that the keystore takes precedence over environment variables:

From https://www.elastic.co/guide/en/logstash/current/keystore.html

When Logstash parses the settings (logstash.yml) or configuration (/etc/logstash/conf.d/*.conf), it resolves keys from the keystore before resolving environment variables.

bhaveshf-cuelogic commented 5 years ago

So, there is no way to pass env variable to logstash config via docker run without declaring it in keystore ? In that case I can't even override it's value from outside since it always prefers keystore value, which is of no use.

I get an error if we try to access any environment variable without adding it to keystore.

jordansissel commented 5 years ago

Oh? My example above only used env vars, not keystore? Can you try that and let me know if it works?

On Thu, Oct 10, 2019 at 2:23 PM Bhavesh Furia notifications@github.com wrote:

So, there is no way to pass env variable to logstash config via docker run without declaring it in keystore ? In that case I can't even override it's value from outside since it always prefers keystore value, which is of no use.

I get an error if we try to access any environment variable without adding it to keystore.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/elastic/logstash/issues/9007?email_source=notifications&email_token=AABAF2SPIIG5Q4W4SCY6BCDQN6MLLA5CNFSM4ENDUNQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA6AJGQ#issuecomment-540804250, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAF2QW6KJKBAUILVK4P6LQN6MLLANCNFSM4ENDUNQQ .

yaauie commented 5 years ago

If the variable exists in the keystore, the value from the keystore is always used; only if it is missing from the keystore is it sourced from the environment.

jordansissel commented 5 years ago

Scenario 2 worked, though — you set the env var to 123 and used it to set the port on tcp input, which errors when trying to bind on that port. Here’s the part of the log showing the port number used:

LogStash::Inputs::Tcp type=>"tcp", port=>123

On Thu, Oct 10, 2019 at 10:31 PM Bhavesh Furia notifications@github.com wrote:

@jordansissel https://github.com/jordansissel : Yes, I tried that too. Following are two different errors when variable is not defined in keystore. CC : @yaauie https://github.com/yaauie

logstash.conf file content:

input { tcp { port => "${MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE}" type => "tcp" } }

Scenario 1 : ENV not defined in keystore and not passed via docker run command

docker run -p 5044:5044 test-build:v1

Error

[2019-10-11T04:53:19,372][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Cannot evaluate ${MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE}. Replacement variable MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE is not defined in a Logstash secret store or as an Environment entry and there is no default value given.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:50:in block in replace_placeholders'", "org/jruby/RubyString.java:3060:ingsub'", "/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:35:in replace_placeholders'", "/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:23:indeep_replace'", "/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:83:in block in config_init'", "org/jruby/RubyHash.java:1419:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:82:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:60:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.3-java/lib/logstash/inputs/tcp.rb:122:in initialize'", "org/logstash/plugins/PluginFactoryExt.java:258:inplugin'", "org/logstash/plugins/PluginFactoryExt.java:120:in buildInput'", "org/logstash/execution/JavaBasePipelineExt.java:50:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

Scenario 2 : ENV not defined in keystore and passed via docker run command

docker run -p 5044:5044 -e MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE=123 test-build:v1

Error

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2019-10-11T05:18:04,805][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [2019-10-11T05:18:04,821][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [2019-10-11T05:18:05,131][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.2"} [2019-10-11T05:18:05,153][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"7764998c-0b3f-4639-bb62-39bd7c235871", :path=>"/usr/share/logstash/data/uuid"} [2019-10-11T05:18:06,265][INFO ][org.reflections.Reflections] Reflections took 43 ms to scan 1 urls, producing 19 keys and 39 values [2019-10-11T05:18:07,515][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2019-10-11T05:18:07,518][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#"} [2019-10-11T05:18:07,785][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"} [2019-10-11T05:18:07,799][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:123", :ssl_enable=>"false"} [2019-10-11T05:18:07,850][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2019-10-11T05:18:07,925][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::Tcp type=>"tcp", port=>123, id=>"1cff6cdccefd739477ce62819c8e3636f12e6333b8a6654d52396d2acada2bbc", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_8e500ffe-2a23-40b7-94f6-4f7690342ffb", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, host=>"0.0.0.0", mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true> Error: Permission denied Exception: Java::JavaNet::SocketException Stack: sun.nio.ch.Net.bind0(Native Method) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:461) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:453)

Also, when I add a new variable to keystore https://www.elastic.co/guide/en/logstash/current/keystore.html#add-keys-to-keystore, it expects me to give a default value, doesn't accept if I set it to null.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/elastic/logstash/issues/9007?email_source=notifications&email_token=AABAF2T2TLVOXHU4G7SR7S3QOAFRPA5CNFSM4ENDUNQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA625QI#issuecomment-540913345, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAF2TAQE3RUO2EKOXU7C3QOAFRPANCNFSM4ENDUNQQ .

bhaveshf-cuelogic commented 5 years ago

Thanks @yaauie and @jordansissel. After reading @yaauie 's reply I removed keystore file from being added in my image and now I am able to access all env variables passed via docker run -e command.

There was a lot of confusion about not being able to access in-sensitive environment variables without adding it to keystore.

bhaveshf-cuelogic commented 5 years ago

Scenario 2 worked, though — you set the env var to 123 and used it to set the port on tcp input, which errors when trying to bind on that port. Here’s the part of the log showing the port number used: LogStash::Inputs::Tcp type=>"tcp", port=>123 … On Thu, Oct 10, 2019 at 10:31 PM Bhavesh Furia @.***> wrote: @jordansissel https://github.com/jordansissel : Yes, I tried that too. Following are two different errors when variable is not defined in keystore. CC : @yaauie https://github.com/yaauie logstash.conf file content: input { tcp { port => "${MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE}" type => "tcp" } } ------------------------------ Scenario 1 : ENV not defined in keystore and not passed via docker run command docker run -p 5044:5044 test-build:v1 Error [2019-10-11T04:53:19,372][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Cannot evaluate ${MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE}. Replacement variable MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE is not defined in a Logstash secret store or as an Environment entry and there is no default value given.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:50:in block in replace_placeholders'", "org/jruby/RubyString.java:3060:ingsub'", "/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:35:in replace_placeholders'", "/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:23:indeep_replace'", "/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:83:in block in config_init'", "org/jruby/RubyHash.java:1419:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:82:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:60:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.3-java/lib/logstash/inputs/tcp.rb:122:in initialize'", "org/logstash/plugins/PluginFactoryExt.java:258:inplugin'", "org/logstash/plugins/PluginFactoryExt.java:120:in buildInput'", "org/logstash/execution/JavaBasePipelineExt.java:50:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]} ------------------------------ Scenario 2 : ENV not defined in keystore and passed via docker run command docker run -p 5044:5044 -e MY_VAR_WHICH_IS_NOT_DEFINED_IN_KEYSTORE=123 test-build:v1 Error OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2019-10-11T05:18:04,805][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [2019-10-11T05:18:04,821][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [2019-10-11T05:18:05,131][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.2"} [2019-10-11T05:18:05,153][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"7764998c-0b3f-4639-bb62-39bd7c235871", :path=>"/usr/share/logstash/data/uuid"} [2019-10-11T05:18:06,265][INFO ][org.reflections.Reflections] Reflections took 43 ms to scan 1 urls, producing 19 keys and 39 values [2019-10-11T05:18:07,515][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2019-10-11T05:18:07,518][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#"} [2019-10-11T05:18:07,785][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"} [2019-10-11T05:18:07,799][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:123", :ssl_enable=>"false"} [2019-10-11T05:18:07,850][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2019-10-11T05:18:07,925][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::Tcp type=>"tcp", port=>123, id=>"1cff6cdccefd739477ce62819c8e3636f12e6333b8a6654d52396d2acada2bbc", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_8e500ffe-2a23-40b7-94f6-4f7690342ffb", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, host=>"0.0.0.0", mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true> Error: Permission denied Exception: Java::JavaNet::SocketException Stack: sun.nio.ch.Net.bind0(Native Method) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:461) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:453) Also, when I add a new variable to keystore https://www.elastic.co/guide/en/logstash/current/keystore.html#add-keys-to-keystore, it expects me to give a default value, doesn't accept if I set it to null. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9007?email_source=notifications&email_token=AABAF2T2TLVOXHU4G7SR7S3QOAFRPA5CNFSM4ENDUNQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA625QI#issuecomment-540913345>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAF2TAQE3RUO2EKOXU7C3QOAFRPANCNFSM4ENDUNQQ .

Yes, I realized that as I posted the message, that's why deleted it immediately

ananthulasrikar commented 5 years ago

@nodesocket - are you running Logstash via the service ?

If so, the service runs as the logstash user which is a no shell user, and thus no shell environment variables (like /etc/environment). To set environment variables for the logstash service you should be able set it like this:

echo "ELASTICSEARCH_ENDPOINT=search-myorg-elasticsearch-dev-gvxi25arm6xjn7y4.us-west-2.es.amazonaws.com" | sudo tee --append /etc/sysconfig/logstash

If this is not from a deb/rpm package, does it work from the bash shell ?

We are using rpm package.May I please know how can we load /etc/environment in such case ?

aruandre commented 3 years ago

placing the env values in /etc/default/logstash does the trick all the other solutions provider here for ubuntu do not work

jasenj1 commented 3 years ago

Yes. /etc/default/logstash and /etc/sysconfig/logstash need to be documented. I just wasted most of a day cursing Logstash because my environment variables weren't being picked up when run as a service. For development, I run Logstash from the command line. Once everything is tuned, I turn it over to the service. Works one way, broken the other. Not good.