logstash_admin builtin role permissions not enough for central pipeline management

[eedugon]

Elasticsearch version (bin/elasticsearch --version): 6.5.4

Plugins installed: [N/A]

JVM version (java -version): N/A

OS version (uname -a if on a Unix-like system): N/A

Description of the problem including expected versus actual behavior: When following the instructions to get centralized pipeline management to work with logstash the following is mentioned:

If your Elasticsearch cluster is protected with basic authentication, assign the logstash_admin role to any users who will use centralized pipeline management

With that role only logstash doesn't start and returns a 403 error coming from Elasticsearch:

[2019-01-16T18:52:58,866][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_admin_user:xxxxxx@my_es_url:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://my_es_url:9243/'"}

Just adding to the user the "cluster:monitor" privilege (monitor privilege at cluster level) it works, so I believe the permission might be missing in logstash_admin role.

Steps to reproduce: Based on this document

1. Create a user for centralized pipeline management and add the builtin role logstash_admin to the user.
2. Enable and configure centralized pipeline management in logstash, with the user created in the previous step in xpack.management.elasticsearch.username setting. Add also the password, the url and set enabled to true.
3. Start logstash, it will fail with a 403 error

Workaround:

1. Stop logstash
2. Create a role in Elasticsearch with privilege "cluster:monitor".
3. Assign that new role to the previously created user.
4. Start logsatsh, it will start properly.

Notes: There's one other document (not aligned with the previously shared one) that includes a different message, mentioning that for central management to work, the roles logstash_admin and the custom role logstash_writer should be added to the user.

My view is that this document is probably not the right solution, because:

• logstash_writer is a custom role, created for a different purpose.
• The permissions in logstash_writer role are more than the minimum permissions needed for the central pipeline management to work.

I have already rasied a PR to align the documents (some documents mention to include the builtin_role, others mention to add both roles), but I believe that it's better to decide first what's the root cause of this and if logstash_admin role is missing a permission or not.

@jsvd is also aware of this.

[elasticmachine]

Pinging @elastic/es-security

[tvernum]

Why does logstash central management need to hit the / endpoint in ES? Is it just using that to test whether the node is available, or does it actually need data from that endpoint?

[bphenriques]

Hitting this issue as well.

Given that:

• The Logstash monitoring and management can communicate with different clusters
• Logstash monitoring uses a dedicated user for that purpose.

I suggest a new "built-in user" logstash_management_ system with the role logstash_admin(as required by the CPM). A better user name would be logstash_admin_system given the name of the role :)

[mjmbischoff]

Why does logstash central management need to hit the / endpoint in ES? Is it just using that to test whether the node is available, or does it actually need data from that endpoint?

@tvernum since its a payed feature it hits _xpack and needs cluster:monitor/xpack/info That still gives issues (reporting 403 but not much more info given default log settings) and giving it the monitoring role makes it work, unsure why it's hitting something else governed by the permissions given by the monitor role (ES 7.15)