401 Not authorized - Githubissues

kfrajtak commented 2 years ago

(check apply)

[X] read the contribution guideline
[x] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

I am not able to connect fluentd to ES with user elastic with password 'changme' (quotes are part of the password).

{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]",
"header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception",
"reason":"unable to authenticate user [elastic] for REST request [/]",
"header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}},"status":401}

REST API call from CLI works:

curl --user "elastic:'changeme'" -X GET http://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 11,
  "active_shards" : 11,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Steps to replicate

Set ES account with given username and password.

Configuration file did not work neither with directly typed password 'changeme' nor with escaped %27changme%27 nor with %{'changeme'}...

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<filter>
  @type parser
  format json      
  key_name log
  emit_invalid_record_to_error false
  remove_key_name_field true
  reserve_data true
</filter>
<match *.**>
  @type elasticsearch
  host elasticsearch
  port 9200
  user elastic
  password %{'changeme'}
  logstash_format true
  logstash_prefix fluentd
  flush_interval 1s
</match>

Expected Behavior or What you need to ask

The question is how to make this work?

Using Fluentd and ES plugin versions

Windows 10
Docker
Fluentd v1.14.5 (docker image fluent/fluentd:v1.14-1)

ES plugin 5.2.1

2022-04-10 08:19:44 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-04-10 08:19:44 +0000 [info]: gem 'fluentd' version '1.14.5'
2022-04-10 08:19:44 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.1'

ES version 8.2.1

cosmo0920 commented 2 years ago

How about using %27changeme%27 in password?

password %27changeme%27


irb> require 'cgi'
irb> CGI.escape("'changeme'")
=> "%27changeme%27"

kfrajtak commented 2 years ago

I tried that as well and it does not work.

But then I gave it a second thought - ES has now more strict security model and maybe it's not just the question of user name and password but it also checks the access to indices etc. So I created a role in ES for fluentd and matching index (relaxing everything else)

And then I created new user granting the user this role and it worked!

Here's a list of links that helped me:

Can you update project homepage with this information?

cosmo0920 commented 2 years ago

Thanks for the information. I'll update README w/ the information.

uken / fluent-plugin-elasticsearch

401 Not authorized #963

Problem

Steps to replicate

Expected Behavior or What you need to ask

Using Fluentd and ES plugin versions