Please explain requirement for s3::DeleteObject IAM permission

[diclophis]

Hello,

We are implementing ES s3 snapshots as a mechanism for secure storage of system logs and we do not desire to allow the search cluster to be able to "delete" anything from the s3 bucket (for auditing purposes).

Is the s3::DeleteObject permission absolutely required? Or can it be disabled via a config switch. We would be OK with having the bucket "fill up" indefinitely while storing snapshots.

Any help / suggestions would be appreciated...

[dadoonet]

Yes. We need to delete unused old files (old segments) at some point.

I'm unsure if we should support such a "do not remove anything" feature. @imotov thoughts? Should we open an issue in core?

[diclophis]

Thanks for the quick reply... Do these "old segment" files have a particularly unique filename signature that we could restrict the s3::DeleteObject by?

[dadoonet]

There can be a lot of files extension.

Just index some data in a cluster, and look at the data dir. Index new data again after some seconds (let say 6 seconds). Look at the data dir again.

Then run an optimize with number of segments = 1 and look at files that have been removed. That could give you an idea. Probably incomplete though.

[diclophis]

Ok... we also discovered another oddity regarding security/permissions...

It seems when you fetch: http://your-es-cluster-host/_snapshot/your-es-s3-backup-bucket-repository the AWS access_key and secret_key (it looks like all settings for the repository in fact) are returned in the HTTP response.

This is extremely undesirable for our use case because of the required s3::DeleteObject permission... meaning if our network is compromised (or even worse, a node in our ES cluster is compromised) the attackers would be able to fetch this URL, gain access to our S3 bucket, and be able to delete the "secure audit trail" we are attempting to create in S3 via this plugin.

Please advise if you can recommend a way to disable this endpoint or an alternate way of configuring this plugin such that the keys are not presented in any HTTP response. And if you recommend using elasticsearch as a "secured, immutable log store/database".

[dadoonet]

Which version are you using?

[diclophis]

Our stack looks like this

elasticsearch  1.4.5
curator  3.3.0
elasticsearch-cloud-aws 2.4.2

Again, thanks for the quick responses...

[dadoonet]

It has been fixed in 2.0.0: https://github.com/elastic/elasticsearch/pull/12845

But it sounds like it was not backported

[dadoonet]

Also related PR in this project: https://github.com/elastic/elasticsearch-cloud-aws/pull/180

[diclophis]

Ok, we were able to configure the secret tokens in our elasticsearch.yml which seems to prevent the key leakage. Thanks again, gonna close this ticket as all of our issues have been addressed

[imotov]

@dadoonet I think it is possible to modify the way snapshots are created to make them write-once. There are only two things that are preventing this at the moment - 1) in order to ensure that we can read snapshots at any time we write snapshot metadata files using a temporary name and then atomically move them into place. 2) we create a single index file for url repository. I think it should be relatively easy to add a switch that would disable both of these behaviors.