search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.2k stars 3.5k forks source link

Logstash throws a "Failed to install template" on events sent from Elastic Agent #14055

Closed dedemorton closed 2 years ago

dedemorton commented 2 years ago

Logstash information:

Please include the following information:

  1. Logstash version: 8.2.0
  2. Logstash installation source: expanded from tar
  3. How is Logstash being run: via command line

Plugins installed: (bin/logstash-plugin list --verbose)

logstash-codec-avro (3.3.1)
logstash-codec-cef (6.2.4)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.0)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.2.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.14)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.1.4)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.11.1)
logstash-filter-fingerprint (3.3.2)
logstash-filter-geoip (7.2.12)
logstash-filter-grok (4.4.1)
logstash-filter-http (1.4.0)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.1.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.3.0)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.1.3)
logstash-input-azure_event_hubs (1.4.3)
logstash-input-beats (6.3.0)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (1.1.11)
logstash-input-elasticsearch (4.12.3)
logstash-input-exec (3.4.0)
logstash-input-file (4.4.0)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.1)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.5.0)
logstash-input-http_poller (5.3.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.1)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-s3 (3.8.3)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-sqs (3.3.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.2.7)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.1)
logstash-integration-elastic_enterprise_search (2.2.1)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.2.4)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (10.10.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq (7.3.0)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-cloudwatch (3.0.10)
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.4.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.5.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-s3 (4.3.5)
logstash-output-sns (4.0.8)
logstash-output-sqs (6.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.0.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.2)

JVM (e.g. java -version):

bundled version

OS version (uname -a if on a Unix-like system):

Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:

Sending events from Elastic Agent to Logstash (with TLS enabled) results in Failed to install template error.

According to @jsvd, this is a bug. Template management should be disabled by default when writing to data streams.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

  1. Follow the steps in this guide to send data from Fleet-managed Elastic Agents to Logstash: https://www.elastic.co/guide/en/fleet/8.2/secure-logstash-connections.html
  2. Notice that the console reports the following error:
[2022-04-22T00:11:04,026][ERROR][logstash.outputs.elasticsearch][elastic-agent-pipeline] Failed 
to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 
'https://58f88fcaeb294e459908dae6e61807a4.us-west2.gcp.elastic-
cloud.com:443/_index_template/ecs-logstash'", 
:exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, 
:backtrace=>["/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client.rb:408:in `template_put'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/plugin_mixins/elasticsearch/common.rb:149:in `block in 
after_successful_connection'"]}
jsvd commented 2 years ago

moved to the ES output plugin repository https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1071 thanks for the report @dedemorton 💟