search

kopeio / etcd-manager

operator for etcd: moved to https://github.com/kubernetes-sigs/etcdadm
Apache License 2.0
164 stars 45 forks source link

etcd-manager backup cleaner calls vfs storage quite heavily #356

Closed zetaab closed 3 years ago

zetaab commented 3 years ago

When we are looking S3 bills we can see that between september - october our S3 bills went 10x. We are now searching the reason for that, and one possible PR for this is https://github.com/kopeio/etcd-manager/pull/328. Increasing the costs 10 times is IMO too much.

I went also through the audit logs and it seems that the backup thing is the problem https://gist.github.com/zetaab/b5e8c2934611bad0ff67b097cd3d9c9a (single cluster making over 700 ListObjects/ListObjectVersions calls in less than 20 seconds. And it does this quite often)

image

This statistics contains 26 kops clusters, so in spikes we see over 70k requests per 5 minutes against bucket. The spike happens each hour and etcd backup cleaner is executed in each hour.

Could we for instance execute this backup cleaner once a day, instead of every hour? Could we make the cleaner interval configurable? Now it costs quite much money.

/kind bug

zetaab commented 3 years ago

seems that the issue was maybe with missing permission s3:ListBucketVersions from the IAM policy