search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.5k stars 24.89k forks source link

Elasticsearch audit logging: Logging the query result #67918

Open arisonl opened 3 years ago

arisonl commented 3 years ago

The need to log the results of a a query has come up a number of times, including in the context of alerting (see linked issue).

cc @bytebilly

elasticmachine commented 3 years ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

bytebilly commented 3 years ago

We currently don't have the ability to get the results related to the original request when writing the audit log entry.

However that would be helpful to track things like the API key ID when it's created (since it's automatically generated by the system and not provided as part of the request). This was discussed some time ago in the ES Security team.

cc @elastic/es-security

rjernst commented 3 years ago

I'm going to relabel this to the security team since it does not appear to involve logging infrastructure, but just the particular use by the security team with their audit log.

albertzaharovits commented 3 years ago

@arisonl I had a quick chat with @tvernum about this.

We believe, the feature to audit responses doesn't fit squarely in the audit log that we've got; for example the audit log contains authentication failures.

The ES slowlog looks like a more suitable option. This log is more geared specifically for search operations, and it already prints the search queries. It's rather shard oriented, so "responses" might not be what is usually expected. Maybe someone from @elastic/es-search can comment about the feasibility of including responses in the slowlog, please?

The slowlog lacks the user context, and this is something we could work on the security side, if this is something we need (but it's not clear if so, maybe you can confirm).

arisonl commented 3 years ago

@albertzaharovits thank you for your response. This request has come up in the context of alerting. Users would like to be able to go back and investigate the raw results returned by an alerting query. I only have a high level understanding of this ES feature, so if I understand correctly, if we were to go down this path, we would need to add this piece of context on top of the ability to record the results and in addition, users would have to piece together the results from the shards, perhaps using this context?