search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine. #6212

Open jpengenheiro opened 5 years ago

jpengenheiro commented 5 years ago

Recently I've observed what I consider to be unexpected behaviour while bootstraping graylog as an unprivileged user.

Here is a link to the troubleshooting I already made:

Expected Behavior

Graylog should create the correct templates, indices and aliases with a user that can only access indices with a cretain prefix, this would allow for easy setup of multitenant ES clusters with several graylog clients.

Current Behavior

while bootstraping as an unprivileged user:

  1. graylog fails to create the graylog-internal template
  2. graylog creates an index suffixed with deflector ex: graylogdev_default_deflector.

while bootstraping as privileged elastic user:

  1. graylog creates the graylog-internal template
  2. graylog creates an index suffixed with a number ex: graylogdev_default_0.

after bootstraping I can switch to an unprivileged user:

  1. index set creation results in the correct creation of a new template
  2. a new index is correctly created, suffixed with a number ex: graylogdev_new_0.

Possible Solution

It would be nice to know exactly what are the strictly required privileges to have graylog using an ES cluster as an unprivileged user.

Steps to Reproduce (for bugs)

The troubleshooting I already made:

https://community.graylog.org/t/unusual-behaviour-while-using-elasticsearch-non-super-user/11355/1

Context

This is preventing the design of a multitenant ES cluster solution, where every user would not be able to access any index not explicitely allow by an administrator.

This was encountered when attempting to implement authentication and authorization for the ES cluster by creating a user with restricted index use permissions.

Environment

jdekoning commented 5 years ago

@jpengenheiro I am very interested in how your configuration now looks like... Whenever I enable x-pack I always end up in your first scenario (fails to create templates and indices, which is actually auto created by ES). To reduce the problem as much as possible I have disabled almost all x-pack features. When I disable it entirely, there are no problems (xpack.security.enabled = "false").

I only have xpack.security.enabled = "true" enabled and pretty much disabled authentication as well: xpack.security.authc.anonymous.roles = "superuser".

Http ssl is still disabled xpack.security.http.ssl.enabled = "false"

At the time of writing I was using: elasticsearch:6.8.2 graylog:3.1

And to avoid the Deflector exists as an index and is not an alias error i have added action.auto_create_index = ".watches,.triggered_watches,.watcher-history-*". Which makes the logs flood with messages like: 2019-09-16 08:37:34,010 WARN : org.graylog2.indexer.messages.Messages - Failed to index message: index= id=<3a767271-d85d-11e9-a848-0242ac110002> error=<{"type":"index_not_found_exception","reason":"no such index and [action.auto_create_index] ([.watches,.triggered_watches,.watcher-history-*]) doesn't match","index_uuid":"_na_","index":"graylog_deflector"}>

To me this issue seems similar, but I can also create a new one if needed.

jdekoning commented 5 years ago

After some poking around I got to the point that I would initialize the graylog-internal template myself. After creating graylog-internal and also creating a first graylog_0 index, everything seems to work fine. Using the free services of x-pack such as authentication and tls on both transport and http for elasticsearch is now not a problem anymore.

So the workaround I found to be working:

curl -X PUT --user <user>:<pass> https://elastic.node:9200/_template/graylog-internal -H 'Content-Type: application/json' -d' {"order":-1,"index_patterns":["graylog_*"],"settings":{"index":{"analysis":{"analyzer":{"analyzer_keyword":{"filter":"lowercase","tokenizer":"keyword"}}}}},"mappings":{"message":{"properties":{"message":{"type":"text","analyzer":"standard","fielddata":false},"full_message":{"type":"text","analyzer":"standard","fielddata":false},"timestamp":{"type":"date","format":"yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"yyyy-MM-dd HH:mm:ss.SSS"},"gl2_processing_timestamp":{"type":"date","format":"yyyy-MM-dd HH:mm:ss.SSS"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"streams":{"type":"keyword"}},"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"_source":{"enabled":true}}},"aliases":{"graylog_deflector": {}}} '

Followed by a initialization of the first alias:

curl -X PUT --user <user>:<pass> https://elastic.node:9200/graylog_0

jalogisch commented 5 years ago

according to this community posting it is working with Elasticsearch 6.7.x

https://community.graylog.org/t/kubernetes-install-of-graylog-not-able-to-create-indexes-in-elasticsearch/12247/2

williamsjasonj commented 5 years ago

This workaround is great! Thanks for sharing until a fix is out.

jpengenheiro commented 5 years ago

@jdekoning I have since moved on to other projects and so I probably won't be pursuing this issue anymore, still if you require any information from me please feel free to ask

Your troubleshooting appears to confirm what I already suspected, the problem arises when graylog attempts to create the templates and first index, I remember I could not see any templates being created if I bootstrapped the cluster with an unpriviliged user.

Yours is a better workaround since it does not require knowledge of the elasticsearch's super user's credentials.

mpiscaer commented 4 years ago

@jdekoning Thank you for the workaround.

jrunu commented 4 years ago

At least part of this issue can be attribute to the behaviour change I described in issue elasticsearch/56027. The gist of it is that when x-pack security is enabled /_all/_alias returns a 404, while returning a 200 with an empty JSON object with x-pack disabled. This throws a wrench into the lookup for the deflectors.

Sidenote: /_alias, which is functionally the same I believe, returns a 200 in both cases.

frantz45 commented 4 years ago

Same issue here. In addition to the @jdekoning 's workaround I had to recalculate index ranges in order to be able to search logs

dbu1986 commented 3 years ago

Same here and therefore adding my observations:

We're currently setting up Graylog 3.3.8 with ES 6.8.12. We also need to make use of Elasticsearch Authentication with different users for Graylog and other applications writing directly to Elasticsearch.

According to @jdekoning's solution, I experienced 2 possible workarounds (in our environment) to proceed without errors:

A) Bootstrap Graylog with elastic Superuser in server.conf, then switch to graylog user and restart Graylog.

B) Manually create 5 objects via Elasticsearch API, then initially starting Graylog.

For the second workaround, I initially tried to only set up the index template and the graylog_0 index (as @jdekoning did) but I still got errors in the graylog-server.log. So these are the 5 Requests I had to send in this order:

  1. curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/_template/graylog-internal -H 'Content-Type: application/json' -d '[CONTENT]'
  2. curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/graylog_0
  3. curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/gl-system-events__0
  4. curl -X PUT -u 'graylog:[PASSWORD]' http://[ES-HOST]:9200/gl-events__0
  5. curl -X POST -u 'graylog:[PASSWORD]' -H 'Content-Type: application/json' "http://[ES-HOST]:9200/_aliases" -d '{ "actions" : [ { "add" : { "index" : "graylog_0", "alias" : "graylog_deflector" } } ] }'

Fun Fact: There was no need to add the aliases gl-events_deflector and gl-system-events_deflector as Graylog just created them without any trouble:

2020-10-23 12:33:24,633 INFO : org.graylog2.indexer.MongoIndexSet - Did not find a deflector alias. Setting one up now.
2020-10-23 12:33:24,638 INFO : org.graylog2.indexer.MongoIndexSet - There is no index target to point to. Creating one now.
2020-10-23 12:33:24,646 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <none> to <gl-events_0>.
2020-10-23 12:33:24,646 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <gl-events_0>.
2020-10-23 12:33:24,689 INFO : org.graylog2.indexer.indices.Indices - Successfully created index template gl-events-template
2020-10-23 12:33:24,896 INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <gl-events_0>.
2020-10-23 12:33:24,908 INFO : org.graylog2.indexer.MongoIndexSet - Index <gl-events_0> has been successfully allocated.
2020-10-23 12:33:24,909 INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <gl-events_deflector> to new index <gl-events_0>.
2020-10-23 12:33:24,952 INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <gl-events_deflector> to index <gl-events_0>.
2020-10-23 12:33:24,994 INFO : org.graylog2.indexer.MongoIndexSet - Did not find a deflector alias. Setting one up now.
2020-10-23 12:33:24,997 INFO : org.graylog2.indexer.MongoIndexSet - There is no index target to point to. Creating one now.
2020-10-23 12:33:25,004 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <none> to <gl-system-events_0>.
2020-10-23 12:33:25,004 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <gl-system-events_0>.
2020-10-23 12:33:25,026 INFO : org.graylog2.indexer.indices.Indices - Successfully created index template gl-system-events-template
2020-10-23 12:33:25,234 INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <gl-system-events_0>.
2020-10-23 12:33:25,241 INFO : org.graylog2.indexer.MongoIndexSet - Index <gl-system-events_0> has been successfully allocated.
2020-10-23 12:33:25,242 INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <gl-system-events_deflector> to new index <gl-system-events_0>.
2020-10-23 12:33:25,288 INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <gl-system-events_deflector> to index <gl-system-events_0>.

The Elasticsearch user graylog is receiving these permissions by role:

curl -X GET -u 'elastic:[PASSWORD]' -H 'Content-Type: application/json' "http://[ES-HOST]:9200/_xpack/security/role/allow_graylog_indices?pretty"
{
  "allow_graylog_indices" : {
    "cluster" : [
      "monitor",
      "manage_index_templates"
    ],
    "indices" : [
      {
        "names" : [
          "graylog*",
          "gl*"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

My xpack settings in elasticsearch.yml:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/[ES-HOST].key
xpack.security.transport.ssl.certificate: certs/[ES-HOST].crt
xpack.security.transport.ssl.certificate_authorities: [ "certs/ca.crt" ]
xpack.security.authc.realms.native1.type: native
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.emit_request_body: true
xpack.security.audit.logfile.events.include: access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted, authentication_success, realm_authentication_failed, connection_granted

My Graylog server.conf:

[]# grep -Ev '#|^\s*$' conf/server.conf
is_master = true
node_id_file = /opt/graylog/conf/node-id
password_secret = [PASSWORD_SECRET]
root_password_sha2 = bb87ef5dea2fdf3816c33e3bf58e5a384b07f7b9c8294565c7bacef6d2152210
root_timezone = Europe/Berlin
bin_dir = bin
data_dir = /opt/graylog/data/graylog
plugin_dir = plugin
http_bind_address = [IP]:9000
http_external_uri = https://[EXT-IP]/
elasticsearch_hosts = http://graylog:[PASSWORD]@[ES-HOST]:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 3
elasticsearch_replicas = 1
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /opt/graylog/data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://[GL-NODE1],[GL-NODE2],[GL-NODE3]/graylog?replicaSet=Graylog-RS01
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32