Verify Elasticsearch data integrity

[frantz45]

Expected Behavior

It would be nice if Graylog could check Elastcisearch indices integrity and notify if one have changed.

Current Behavior

At the moment we can't know if someone modified an Elasticsearch index.

Possible Solution

When index rotation happens you could calculate the hash of the index and then you can verify this hash frequently (daily ?). We also need to know if an index have been deleted by something other than the retention policy. It would be nice if a notification would be sent in the Graylog interface, but we also need Graylog generates a log so we could use Graylog functions to be alerted (for example by sending an email, or creating a ticket).

Context

When you work with logs it's important to be sure they have not been modified. For example an attacker could modify logs in order to not be detected. Moreover the law in many countries says we need to retain logs for a certain period, and it's only useful if we can prove logs have not been modified. I posted on the community a long time ago but I don't see any news even in the entreprise version: https://community.graylog.org/t/data-integrity-checking/3090

Your Environment

• Graylog Version: 3.0
• Elasticsearch Version: 6.7
• MongoDB Version: 4
• Operating System: CentOS 7

[gimmic]

You don't really modify existing inserted records in elasticsearch. You would have to delete and reindex, which is not trivial. You can also close or make an index read-only.

It sounds like what you're really wanting is a forensic record of event data. I would recommend a compressed offline copy of the data and validation through a checksum.

I do agree a more formal audit record of graylog would be nice.

[frantz45]

I already have a compressed offline copy of the data and validation through a checksum. The read-only option is a good suggestion. Could Graylog add the option to make the index read-only after rotation ? I have found:

• PUT index_name/_settings {"index":{"blocks.read_only": true}} to make it read-only

• PUT _all/_settings {"index.blocks.read_only_allow_delete": true} to still be able to delete the index after the retention period