untergeek
commented
7 years ago Just for the sake of hare-brained ideas...
What happens if instead of
$ export ES_HOSTS="\"localhost:9200\", \"host2:9200\", \"host3:9200\""you instead did
$ export ES_HOSTS='["localhost:9200", "host2:9200", "host3:9200"]'(of course, you could just escape the brackets instead of using single quotes)
Then you see (and I just verified this in a terminal):
$ echo $ES_HOSTS
["localhost:9200", "host2:9200", "host3:9200"]In theory, this may let you do:
output {
elasticsearch {
hosts => ${ES_HOSTS}
}
}I know that this works in Curator. It may or may not work in Logstash. I figure it's worth a try, though. I think that the ENV variables are parsed first, into strings, which would potentially make the square brackets work when the config is parsed (after env var expansion).
berglh
jordansissel
alesnav
patrickwallaws
fbaligand
nick-george
huyqut
jordanhenderson
marfedd
yaauie
kyrias
I've been replacing the use of
logstash-filter-environmentplugin with the new feature: Using Environment Variables in the Configuration.I initially incorrectly diagnosed and reported the issue here ES hosts array: Support for parsing Environment Variables. @jordansissel advised I should file an issue in the correct project for this particular feature.
One of the things I am attempting to do is to supply an array for configuration item such as the hosts array in the
logstash-output-elasticsearchplugin. This is useful when the environment hostnames and IPs are ephemeral and the use of a service discovery via query ofetcdto determine IP and ports of online elasticsearch nodes during logstash startup is useful.I currently achieve this using sed to replace a placeholder string in the config on the docker container startup. It's not a major issue, but considering environment parsing functionality is built into Logstash now, it'd be great to leverage this instead of hacking the config prior to launch.
Sending Logstash's logs to /home/somebody/logstash-5.0.2/logs which is now configured via log4j2.properties The stdin plugin is now waiting for input: [2016-12-06T16:07:20,738][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://localhost:9200"]}} [2016-12-06T16:07:20,740][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil} [2016-12-06T16:07:20,781][WARN ][logstash.outputs.elasticsearch] Marking url as dead. {:reason=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)", :url=>#<URI::HTTP:0x41764dd8 URL:http://localhost:9200>, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
user@computer:~/logstash-5.0.2$ export ES_HOSTS="\"localhost:9200\", \"host2:9200\", \"host3:9200\""
user@computer:~/logstash-5.0.2$ echo $ES_HOSTS "localhost:9200", "host2:9200", "host3:9200"
user@computer:~/logstash-5.0.2$ bin/logstash -e 'output { elasticsearch { hosts => [ "${ES_HOSTS}" ] } }' Sending Logstash's logs to ~/logstash-5.0.2/logs which is now configured via log4j2.properties The stdin plugin is now waiting for input: [2016-12-06T16:17:02,070][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Host '"localhost:9200", "host2:9200", "host3:9200"' was specified, but is not valid! Use either a full URL or a hostname:port string!>, :backtrace=>["~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:183:in
host_to_url'", "~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:93:inbuild_pool'", "org/jruby/RubyArray.java:2414:inmap'", "~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:93:inbuild_pool'", "~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:20:ininitialize'", "~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:53:inbuild'", "~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch.rb:188:inbuild_client'", "~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:13:inregister'", "~/logstash-5.0.2/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:inregister'", "~/logstash-5.0.2/logstash-core/lib/logstash/output_delegator.rb:37:inregister'", "~/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:196:instart_workers'", "org/jruby/RubyArray.java:1613:ineach'", "~/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:196:instart_workers'", "~/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:153:inrun'", "~/logstash-5.0.2/logstash-core/lib/logstash/agent.rb:250:in `start_pipeline'"]} [2016-12-06T16:17:02,086][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2016-12-06T16:17:05,074][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}user@computer:~/logstash-5.0.2$ bin/logstash -e 'output { elasticsearch { hosts => [ "localhost:9200", "host2:9200", "host3:9200" ] } }' Sending Logstash's logs to ~/logstash-5.0.2/logs which is now configured via log4j2.properties The stdin plugin is now waiting for input: [2016-12-06T16:17:21,387][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://localhost:9200", "http://host2:9200", "http://host3:9200"]}} [2016-12-06T16:17:21,389][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil} [2016-12-06T16:17:21,444][WARN ][logstash.outputs.elasticsearch] Marking url as dead. {:reason=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)", :url=>#<URI::HTTP:0x51dd94f4 URL:http://localhost:9200>, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} [2016-12-06T16:17:21,445][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} [2016-12-06T16:17:21,446][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["localhost:9200", "host2:9200", "host3:9200"]} [2016-12-06T16:17:21,447][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000} [2016-12-06T16:17:21,449][INFO ][logstash.pipeline ] Pipeline main started [2016-12-06T16:17:21,467][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2016-12-06T16:17:26,390][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x588852e4 URL:http://localhost:9200>, :healthcheck_path=>"/"} ^C[2016-12-06T16:17:29,147][WARN ][logstash.runner ] SIGINT received. Shutting down the agent. [2016-12-06T16:17:29,151][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}