Support specify the annotations for SAs

[clyang82]

We have a case to support AWS STS. It is supported by thanos

STS Endpoint

If you want to use IAM credential retrieved from an instance profile, Thanos needs to authenticate through AWS STS. For this purposes you can specify your own STS Endpoint.

By default Thanos will use endpoint: https://sts.amazonaws.com/ and AWS region coresponding endpoints.

In order to support in STS/ROSA clusters, I need to add annotated for SAs: "thanos-store-shard", "thanos-compact", "thanos-receive" , "thanos-receive-controller" to provide ARN Permissoin Policy.

[metalmatze]

That should be possible with jsonnet by extend (merging) the annotations on the ServiceAccounts that kube-thanos exposes. I don't think it's generic enough to add it for everybody, is it?

[clyang82]

Yes. it is not a generic enough. but right now, kube-thanos does not expose the ServiceAccounts. for example: https://github.com/thanos-io/kube-thanos/blob/6328583a623765ed6ebf18064a301104def57420/jsonnet/kube-thanos/kube-thanos-rule.libsonnet#L99-L107 we can expose it so that it can be extendable.

[metalmatze]

Pretty sure that that should be exposed from the function. Since the ruler isn't part of the example.jsonnet I used the store instead:

diff --git a/example.jsonnet b/example.jsonnet
index 95dcebd..b5ef2da 100644
--- a/example.jsonnet
+++ b/example.jsonnet
@@ -47,7 +47,15 @@ local r = t.receiveRouter(commonConfig.config {
 local s = t.store(commonConfig.config {
   replicas: 1,
   serviceMonitor: true,
-});
+}) + {
+  serviceAccount+: {
+    metadata+: {
+      annotations+: {
+        foo: 'bar',
+      },
+    },
+  },
+};

 local q = t.query(commonConfig.config {
   replicas: 1,

The resulting YAML:

diff --git a/manifests/thanos-store-serviceAccount.yaml b/manifests/thanos-store-serviceAccount.yaml
index 823a1cd..fb43929 100644
--- a/manifests/thanos-store-serviceAccount.yaml
+++ b/manifests/thanos-store-serviceAccount.yaml
@@ -1,6 +1,8 @@
 apiVersion: v1
 kind: ServiceAccount
 metadata:
+  annotations:
+    foo: bar
   labels:
     app.kubernetes.io/component: object-store-gateway
     app.kubernetes.io/instance: thanos-store

[clyang82]

Thanks @metalmatze for your example. Yes. it can be done in kube-thanos side. But I cannot use the same way in observatorium-operator. As you know that observatorium-operator depends on observatorium --> thanos --> kube-thanos. It may not a problem in kube-thanos. Anyway, let me handle it by updating the operator manifests.