401 not authorized

[marcofranssen]

Please see the config I'm using to authenticate. For some bizarre reason it just is not able to login.

I also logged the issue over here https://github.com/fluent/fluentd-kubernetes-daemonset/issues/15.

looking forward to a solution.

2017-05-03T15:08:36.192683683Z 2017-05-03 15:08:36 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-05-03 15:09:06 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Unauthorized" error="[401] " plugin_id="object:2b1bd5d5c250"

I do have the following config in place

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v0.12-elasticsearch
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
          limits:
            memory: 200Mi
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        - name: FLUENT_ELASTICSEARCH_USER
          value: "elastic"
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          value: "changeme"
        volumeMounts:
        - name: fluentconfig
          mountPath: /home/fluent/fluentd/etc
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluentconfig
        configMap:
          name: fluent-config

In order to handle the FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD I have added the following configmap.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-config
  namespace: kube-system
data:
  fluent.conf: |
    @include kubernetes.conf

    <match **>
      type elasticsearch
      log_level info
      include_tag_key true
      host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
      port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
      user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
      password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
      logstash_format true
      buffer_chunk_limit 2M
      buffer_queue_limit 32
      flush_interval 5s
      max_retry_wait 30
      disable_retry_limit
      num_threads 8
    </match>
  kubernetes.conf: |
    <match fluent.**>
      type null
    </match>

    <source>
      type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      tag kubernetes.*
      format json
      read_from_head true
    </source>

    ..................
    ................
    .... remainder of kubernetes.conf left for brevity

Is there anyone else experiencing this issue, or knows how to handle the BASIC auth properly?

[marcofranssen]

Any update on this one?

[repeatedly]

Without k8s, you can send data to ES with above config?

[repeatedly]

Ah, eduardo has already committed the patch: https://github.com/fluent/fluentd-kubernetes-daemonset/issues/15#issuecomment-303772297

[ianseyer]

I am now getting this same error, outside of the k8s plugin.

Running td-agent on Amazon Linux.

td-agent conf:

<match debug.**>
  type stdout
</match>

<source>
  type forward
</source>

<source>
  type http
  port 8888
</source>

## live debugging agent
<source>
  type debug_agent
  bind 127.0.0.1
  port 24230
</source>

#
# SYSLOG
#
# get logs from syslog
<source>
  @type syslog
  port 42185
  tag syslog
</source>

<match **>
  @type elasticsearch
  host 127.0.0.1
  port 9200
  logstash_format true
  index_name kibana
  flush_interval 10s # for testing
</match>

Running td-agent version 0.12.36

Error:

2017-07-03 12:02:43 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-07-03 20:46:02 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Unauthorized" error="[401] " plugin_id="object:3ff888e47198"

[cosmo0920]

Which elasticsearch-ruby version do you use? https://github.com/elastic/elasticsearch-ruby#compatibility

[dasterintempus]

I appear to be having a similar issue:

2017-09-18 03:36:09 -0600 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-18 03:36:11 -0600 error_class="Elasticsearch::Transport::Transport::Errors::Unauthorized" error="[401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}" plugin_id="output_elastic_forward"

Config:

<match **>
    @id "output_elastic_#{ENV['fluentname']}"
    @type elasticsearch

    host host.name
    user elastic
    password changeme

    logstash_format true
    logstash_prefix fluent

    buffer_type file
    buffer_path "/a/buffer/path/#{ENV['fluentname']}.elastic*"
</match>

My config does have a valid username and password specified. This issue appeared to start sometime after the output had been running for a day or so. Also noteworthy was that it was using a very high amount of memory in that td-agent process, ~3 GB. The process did not respond to SIGTERM and I had to use SIGKILL to get it to close.

Gem Versions: elasticsearch-5.0.4 elasticsearch-api-5.0.4 elasticsearch-transport-5.0.4 fluent-plugin-elasticsearch-1.9.7 fluentd-0.12.36

[cosmo0920]

My config does have a valid username and password specified.

Your config contains non-alphabetical characters? If so, the following output should be used there:

require 'uri'
URI.encode_www_form_component("changeme")

[dasterintempus]

The config does not contain any characters in need of URI escaping, it's a demo stack, the password really is "changeme". Also this issue appears to only occur after 1-2 days of the plugin successfully outputting data, so I suspect it's some kind of memory leak or other issue.

[cosmo0920]

Is there any suspicious warnings or errors in your elasticsearch log?

[360d-io-labs]

Same problem here, stopped to work after 24-30 hours and I get this in ES logs: FORBIDDEN by default req={ ID:1457884328-727212744#1153490, TYP:BulkRequest, USR:[no basic auth header], BRS:true, ACT:indices:data/write/bulk,

There is nothing in the td-agent.log except the message saying that the access is not authorized. It looks like the credential are somehow lost.

If I restart the fluend process everything is working fine again.

[ferbar]

_reloadconnections queries http://elkserver/_nodes for a list of available ELK nodes after 10k requests. This list doesn't contain the basic auth passwords, so the credentials are lost in the fluentd-elasticsearch plugin after these 10k requests. Solution: never reload configuration:

    reload_connections false
    reload_on_failure false

[cosmo0920]

Thanks for investigating this issue. If your investigatement is true, this is not this plugin’s issue. Could you report lack of basic auth passwords credential within reload_connection query issue in https://github.com/elastic/elasticsearch-ruby?

[cosmo0920]

I investigated this issue. I think that this is elasticsearch-ruby's credential handling issue. see: https://github.com/elastic/elasticsearch-ruby/issues/464#issuecomment-356500993

[fujimotos]

This is a follow-up comment. I believe your "authentication failures after reloading connections" problem has been addressed by the pull request #394.

How to fix the issue

1. Update fluent-plugin-elasticsearch to the version which contains the fix (v2.8.6 or later)
2. If you embedded the login credential in the URL, migrate the settings to use the user and password field instead.

How to migrate your settings

If you have embedded the user/password into the URL as follows:

<match fluent.log>
  @type elasticsearch
  hosts https://admin:secret@server1:443/elastic/
  ...
</match>

Migrate the configuration to the format shown below:

<match fluent.log>
  @type elasticsearch
  hosts server1:443
  scheme https
  user admin
  password secret
  ...
</match>

By migrating to this format, your login credentials should be retained across sessions.

[cosmo0920]

@marcofranssen 's confguration has already used user ... password ... style. He can simply update ES plugin.