Error `Elasticsearch::Transport::Transport::Errors::Forbidden` error 403

[gayan415]

Hey, I used fluentd-es.yaml to pass my kubernetes cluster logs to AWS elasticsearch. But I'm getting an error.

2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/fluentd-elasticsearch-0q0wc_kube-system_fluentd-elasticsearch-9f3081cd99b551408dc128104aa1610be7a973d3e3475d2e8491ab60d47e9f3d.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/bln-portal-ctr-ds-pkkwf_default_bln-cluster-kube-daemonset-d81acd4ef82c1b166e0243e274f11b62b4157bc627c928299ec5f9285fd31141.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/kubernetes-dashboard-3313488171-5w2dt_kube-system_kubernetes-dashboard-1aab181d0508b444df4cc3f5b4928781ddfa00d35b1528d94d97199f666a148a.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/kube-dns-2712020956-76j32_kube-system_dnsmasq-ac3653cdcc7d632bf0e0b3cae92e9a532e200b205087317702028364559a505c.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/kube-proxy-ip-10-0-31-51.ca-central-1.compute.internal_kube-system_kube-proxy-b77331f0cd72bb0136cbf97f1fac143b2f1c19617f662601a6e1c96e87bfaf7c.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/kube-dns-2712020956-76j32_kube-system_kubedns-649550dcc1b92c060d5a0124f917801b5d9c3e68947cc553a2751b44e8aae026.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/kube-dns-2712020956-76j32_kube-system_sidecar-d51407d92451ca39dfc067a0e8a82d099105ca9c1e5883355705f246d310bb5e.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/bln-portal-ctr-ds-pkkwf_default_bln-cluster-kube-daemonset-1b3d56ce94591a4bed3794218d3a1420132cbde70ed7396881a793c6a076e632.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/kube-dns-autoscaler-641111686-wgzjv_kube-system_autoscaler-a20fac6d52d5666156efcbf02226ed6ee40e428f4a6b2d116171dfae687e0a5f.log 2017-09-26 12:49:36 +0000 [info]: following tail of /var/log/containers/bln-portal-ctr-deployment-3803776156-1hvc7_qc_bln-portal-ctr-8aa355ee6df63f6d4652cc7a5332fe30a96d9aad6fd1a857b90b2cc043074105.log 2017-09-26 12:49:42 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-26 12:49:43 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1ea715c" 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:201:in__raise_transport_error' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:312:in perform_request' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/faraday.rb:20:inperform_request' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in perform_request' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/ping.rb:20:inping' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:122:in client' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:279:inrescue in send' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:277:in send' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:271:inwrite' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:345:in write_chunk' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:324:inpop' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:329:in try_flush' 2017-09-26 12:49:42 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:140:inrun' 2017-09-26 12:49:43 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-26 12:49:45 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1ea715c" 2017-09-26 12:49:43 +0000 [warn]: suppressed same stacktrace 2017-09-26 12:49:45 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-26 12:49:49 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1ea715c" 2017-09-26 12:49:45 +0000 [warn]: suppressed same stacktrace 2017-09-26 12:49:49 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-26 12:49:57 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1ea715c" 2017-09-26 12:49:49 +0000 [warn]: suppressed same stacktrace 2017-09-26 12:49:57 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-26 12:50:12 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1ea715c" 2017-09-26 12:49:57 +0000 [warn]: suppressed same stacktrace 2017-09-26 12:50:12 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-26 12:50:42 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1ea715c" 2017-09-26 12:50:12 +0000 [warn]: suppressed same stacktrace 2017-09-26 12:50:42 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-26 12:51:12 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1ea715c" 2017-09-26 12:50:42 +0000 [warn]: suppressed same stacktrace`

Do you have any idea why am I getting this error?

Thank you

[travertischio]

I am getting the same error

@thegayanj Did you find a solution?

[mmadsen]

I've got this too (after getting past RBAC errors). Hoping someone has a diagnosis or pointer!

[tobiasbrodersen]

403 followed only by the plugin ID means that it could connect to the es cluster, check your url, id, secret and zone again. you can see in the logs what is tries to connect to and with what.