[Bug] Operator randomly tries to scale up and down Grafana operated service

ivan-kovnatskyi-c54f97a commented 2 months ago

Describe the bug Grafana operator tries to constantly reconcile Grafana deployment resulting with unresponsive grafana service, scaling up and down randomly the service, creating, deleting replicasets.

Version v5.12.0

To Reproduce Steps to reproduce the behavior:

Configure dashboards or datasource resources using plugins (as described in https://grafana.github.io/grafana-operator/docs/examples/plugins/readme/)
Use different versions of the same plugin across multiple resources
Observe the operator's behavior

Expected behavior Grafana operator checks the current service and declared code to deploy that, sees no difference and does nothing, when there's no change in kind: Grafana.

Suspect component/Location where the bug might be occurring Various plugin versions across dashboards.

If you're using plugins in your dashboards/datasource resource (like https://grafana.github.io/grafana-operator/docs/examples/plugins/readme/) and they are not on the same version, this will cause the operator to flip between versions. Can you check that this is not the case?

Screenshots If applicable, add screenshots to help explain your problem.

Runtime (please complete the following information):

OS: Linux, k8s, , bottlerocket
Grafana Operator Version: 5.12.0
Environment: Kubernetes, eks, 1.30
Deployment type: kustomize, flux

Additional context

Grafana config:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
 name: grafana-operated
 namespace: monitoring
 labels:
  dashboards: "grafana"
spec:
 client:
  preferIngress: false
  timeout: 15
 persistentVolumeClaim:
  spec:
   accessModes: [ReadWriteOnce]
   resources:
    requests:
     storage: 1Gi
   storageClassName: gp3
 deployment:
  spec:
   strategy:
    type: Recreate
   replicas: 1
   template:
    spec:
     containers:
      - name: grafana
       image: grafana/grafana-oss:11.1.4-ubuntu
       envFrom:
        - secretRef:
          name: "grafana-auth-credentials-${tf_eks_cluster_name}"
       resources:
        limits:
         memory: 1Gi
        requests:
         cpu: 500m
         memory: 1Gi
     volumes:
      - name: grafana-data
       persistentVolumeClaim:
        claimName: grafana-operated-pvc
     nodeSelector:
      scope: "monitoring"
     securityContext:
      fsGroup: 472
      supplementalGroups:
       - 0
     tolerations:
      - key: "Monitoring"
       operator: "Equal"
       value: "true"
       effect: "NoSchedule"
 service:
  metadata:
   labels:
    prometheus.io/scrape: "true"
 ingress:
  metadata:
   annotations:
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /api/health
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
  spec:
   ingressClassName: "operations"
   rules:
    - host: "grafana.${tf_operations_domain}"
     http:
      paths:
       - backend:
         service:
          name: grafana-operated-service
          port:
           number: 3000
        path: /
        pathType: Prefix
 config:
  log:
   mode: "console"
  dataproxy:
   timeout: "60"
  analytics:
   reporting_enabled: "false"
   check_for_updates: "false"
  users:
   auto_assign_org_role: "Viewer"
   viewers_can_edit: "true"
  auth:
   disable_login_form: "true"
   disable_signout_menu: "true"
  auth.okta:
   auto_login: "true"
   name: "Okta"
   enabled: "true"
   scopes: "openid,profile,email,groups"
   allow_sign_up: "true"
   api_url: "[redacted]"
   auth_url: "[redacted]"
   token_url: "[redacted]"
   role_attribute_strict: "true"
   allowed_groups: ""
   role_attribute_path: "appRole"
  metrics:
   enabled: "true"

Operator install config:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
 name: grafana-operator
 namespace: monitoring
spec:
 interval: 30m
 url: "oci://ghcr.io/grafana/helm-charts/grafana-operator"
 ref:
  tag: "v5.12.0"

Operator logs:

2024-08-28T08:52:37Z    ERROR   GrafanaReconciler   reconciler error in
stage   {"controller": "grafana", "controllerGroup": "grafana.integreatly.org",
"controllerKind": "Grafana", "Grafana":
{"name":"grafana-operated","namespace":"monitoring"}, "namespace":
"monitoring", "name": "grafana-operated", "reconcileID":
"94120e52-dd09-4031-ba9b-3dbbaada1cdd", "stage": "service", "error": "client
rate limiter Wait returned an error: context canceled"}

2024-08-28T08:52:37Z    ERROR   Reconciler error    {"controller":
"grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind":
"Grafana", "Grafana": {"name":"grafana-operated","namespace":"monitoring"},
"namespace": "monitoring", "name": "grafana-operated", "reconcileID":
"bc2850af-fe9f-4693-bdc5-5582669f6dca", "error": "Operation cannot be fulfilled
on grafanas.grafana.integreatly.org \"grafana-operated\": the object has been
modified; please apply your changes to the latest version and try again"}

2024-08-28T08:52:37Z    INFO    Warning: Reconciler returned both a non-zero
result and a non-nil error. The result will always be ignored if the error is
non-nil and the non-nil error causes reqeueuing with exponential backoff. For
more details, see:
https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler
{"controller": "grafana", "controllerGroup": "grafana.integreatly.org",
"controllerKind": "Grafana", "Grafana":
{"name":"grafana-operated","namespace":"monitoring"}, "namespace":
"monitoring", "name": "grafana-operated", "reconcileID":
"bc2850af-fe9f-4693-bdc5-5582669f6dca"}

2024-08-28T08:52:37Z    ERROR   GrafanaReconciler   failed to get version
from instance   {"controller": "grafana", "controllerGroup":
"grafana.integreatly.org", "controllerKind": "Grafana", "Grafana":
{"name":"grafana-operated","namespace":"monitoring"}, "namespace":
"monitoring", "name": "grafana-operated", "reconcileID":
"bc2850af-fe9f-4693-bdc5-5582669f6dca", "error": "fetching version: Get
\"http://grafana-operated-service.monitoring:3000/api/frontend/settings\":
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"}

2024-08-28T08:52:22Z    ERROR   GrafanaReconciler   failed to get version
from instance   {"controller": "grafana", "controllerGroup":
"grafana.integreatly.org", "controllerKind": "Grafana", "Grafana":
{"name":"grafana-operated","namespace":"monitoring"}, "namespace":
"monitoring", "name": "grafana-operated", "reconcileID":
"63d7cb28-3aa4-41d5-b0c6-1a814227d534", "error": "fetching version: Get
\"http://grafana-operated-service.monitoring:3000/api/frontend/settings\": dial
tcp 172.20.138.193:3000: connect: connection refused"}

Grafana-operated-deployment describe:

│ Conditions:                                                                                          │
│  Type      Status Reason                                                                                │
│  ----      ------ ------                                                                                │
│  Progressing  True  NewReplicaSetAvailable                                                                        │
│  Available   True  MinimumReplicasAvailable                                                                       │
│ OldReplicaSets: grafana-operated-deployment-779fc75d95 (0/0 replicas created), grafana-operated-deployment-78488687f (0/0 replicas created), grafana-operated-deployment-858d55865c (0/0 repl │
│ icas created), grafana-operated-deployment-fb6c9ff47 (0/0 replicas created), grafana-operated-deployment-7df69f9469 (0/0 replicas created)                           │
│ NewReplicaSet:  grafana-operated-deployment-c69d6557c (1/1 replicas created)                                                         │
│ Events:                                                                                            │
│  Type  Reason       Age          From          Message                                                       │
│  ----  ------       ----          ----          -------                                                       │
│  Normal ScalingReplicaSet 25m (x284 over 17h)  deployment-controller Scaled up replica set grafana-operated-deployment-858d55865c to 1 from 0                      │
│  Normal ScalingReplicaSet 25m (x299 over 17h)  deployment-controller Scaled down replica set grafana-operated-deployment-858d55865c to 0 from 1                     │
│  Normal ScalingReplicaSet 24m (x549 over 17h)  deployment-controller Scaled up replica set grafana-operated-deployment-7df69f9469 to 1 from 0                      │
│  Normal ScalingReplicaSet 24m (x541 over 17h)  deployment-controller Scaled down replica set grafana-operated-deployment-7df69f9469 to 0 from 1                     │
│  Normal ScalingReplicaSet 7m11s (x564 over 17h) deployment-controller Scaled up replica set grafana-operated-deployment-78488687f to 1 from 0                       │
│  Normal ScalingReplicaSet 7m11s (x561 over 17h) deployment-controller Scaled down replica set grafana-operated-deployment-78488687f to 0 from 1

Related threads on Grafana Slack:

Regarding plugins, we have these:

.withDashboards((import
'grafana/cardinality-dashboards.libsonnet').grafanaDashboards, 'Mimir', [{
    name: 'marcusolsson-treemap-panel', version: '2.0.1' }])

local mimirJsonApiDatasourceResource(cluster) = {
  apiVersion: 'grafana.integreatly.org/v1beta1',
  kind: 'GrafanaDatasource',
  metadata: {
    name: std.format('mimir-json-api-%s', cluster),
  },
  spec: {
    instanceSelector: {
      matchLabels: {
        dashboards: 'grafana',
      },
    },
    datasource: {
      access: 'proxy',
      type: 'marcusolsson-json-datasource',
      jsonData: {
        tlsSkipVerify: true,
      },
      name: std.format('Mimir JSON API %s', cluster),
      url: 'http://mimir-gateway.mimir/prometheus/api/v1/cardinality/',
    },
    plugins: [
      {
        name: 'marcusolsson-json-datasource',
        version: '1.3.9',
      },
    ],
  },
};

.withDashboards((import
'grafana/kubernetes-worker-node-capacity.libsonnet').grafanaDashboards,
'Kubernetes', [{ name: 'digrich-bubblechart-panel', version: '2.0.1' }])

Notable mention would be is that we also periodically sync GrafanaDashboards using operator:

local dashboardResource(name, json, folder, plugins) = {
  apiVersion: 'grafana.integreatly.org/v1beta1',
  kind: 'GrafanaDashboard',
  metadata: {
    name: std.strReplace(name, '.json', '') + '-dashboard',
  },
  spec: {
    plugins: plugins,

    ///////////////////
    resyncPeriod: '2m',
    ///////////////////

    folder: folder,
    instanceSelector: {
      matchLabels: {
        dashboards: 'grafana',
      },
    },
    json: |||
      %s
    ||| % [json],
  },
};

theSuess commented 2 months ago

Can you add the yaml representation of the replica sets to the issue?

e.g:

kubectl get replicasets/grafana-operated-deployment-78488687f -o yaml

for all replica sets in the deployment describe

ivan-kovnatskyi-c54f97a commented 2 months ago

Sure:

───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: grafana-operated-deployment-7df69f9469.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ apiVersion: apps/v1
   2   │ kind: ReplicaSet
   3   │ metadata:
   4   │   annotations:
   5   │     deployment.kubernetes.io/desired-replicas: "1"
   6   │     deployment.kubernetes.io/max-replicas: "1"
   7   │     deployment.kubernetes.io/revision: "6061"
   8   │     deployment.kubernetes.io/revision-history: 4582,4589,4594,4596,4599,4603,4606,4608,4610,4612,4615,4617,4619,4624,4628,4637,4639,4641,4645,4647,4654,4657,4668,4671,4673,4683,4685,468
       │ 7,4692,4695,4701,4703,4707,4711,4714,4716,4719,4725,4727,4729,4738,4744,4753,4758,4760,4763,4767,4772,4774,4781,4784,4788,4791,4799,4801,4805,4807,4809,4811,4817,4824,4827,4833,4835,484
       │ 1,4845,4854,4856,4858,4863,4868,4879,4881,4885,4888,4890,4892,4894,4899,4904,4906,4909,4912,4918,4922,4924,4928,4931,4934,4940,4943,4947,4949,4951,4953,4956,4958,4961,4970,4973,4975,497
       │ 8,4982,4984,4989,4991,4997,5002,5005,5007,5011,5013,5017,5021,5024,5026,5029,5031,5034,5036,5041,5044,5046,5049,5051,5053,5058,5061,5065,5070,5072,5074,5079,5085,5087,5090,5093,5101,510
       │ 8,5114,5116,5124,5129,5137,5146,5148,5150,5156,5158,5163,5168,5172,5174,5176,5178,5184,5188,5190,5196,5200,5202,5210,5218,5221,5225,5227,5229,5231,5238,5242,5246,5248,5251,5253,5260,526
       │ 5,5267,5274,5276,5279,5282,5285,5289,5299,5302,5311,5314,5320,5323,5325,5329,5333,5337,5339,5343,5348,5351,5353,5361,5363,5366,5368,5370,5375,5377,5382,5385,5390,5392,5394,5396,5399,540
       │ 5,5407,5410,5412,5416,5421,5423,5425,5427,5429,5431,5433,5435,5439,5442,5444,5447,5449,5453,5456,5458,5462,5468,5470,5473,5478,5480,5482,5484,5486,5491,5497,5499,5504,5506,5508,5510,551
       │ 4,5518,5520,5523,5525,5527,5530,5534,5537,5539,5541,5549,5557,5565,5574,5582,5584,5586,5590,5592,5598,5604,5610,5612,5614,5623,5627,5633,5636,5639,5642,5644,5647,5650,5652,5655,5658,566
       │ 2,5664,5667,5669,5671,5674,5680,5683,5686,5688,5690,5692,5695,5701,5703,5717,5724,5729,5732,5734,5736,5741,5746,5748,5750,5754,5759,5761,5764,5769,5771,5773,5776,5779,5782,5784,5788,579
       │ 0,5794,5799,5809,5811,5814,5817,5819,5821,5824,5830,5832,5834,5836,5839,5841,5844,5848,5850,5852,5854,5858,5861,5867,5873,5881,5885,5887,5889,5895,5904,5906,5910,5912,5915,5920,5923,592
       │ 8,5934,5938,5946,5949,5952,5954,5957,5962,5964,5966,5968,5970,5972,5974,5976,5980,5982,5988,5990,5992,5995,5997,5999,6003,6007,6011,6014,6017,6019,6024,6026,6034,6038,6040,6042,6045,604
       │ 9,6051,6054
   9   │   creationTimestamp: "2024-08-27T10:00:14Z"
  10   │   generation: 1575
  11   │   labels:
  12   │     app: grafana-operated
  13   │     pod-template-hash: 7df69f9469
  14   │   name: grafana-operated-deployment-7df69f9469
  15   │   namespace: monitoring
  16   │   ownerReferences:
  17   │   - apiVersion: apps/v1
  18   │     blockOwnerDeletion: true
  19   │     controller: true
  20   │     kind: Deployment
  21   │     name: grafana-operated-deployment
  22   │     uid: d7edab22-c6f6-4c63-8fd4-d2fa8a0b798f
  23   │   resourceVersion: "633407289"
  24   │   uid: 008ae2f1-2fb2-4515-923e-5fdff907d970
  25   │ spec:
  26   │   replicas: 1
  27   │   selector:
  28   │     matchLabels:
  29   │       app: grafana-operated
  30   │       pod-template-hash: 7df69f9469
  31   │   template:
  32   │     metadata:
  33   │       creationTimestamp: null
  34   │       labels:
  35   │         app: grafana-operated
  36   │         pod-template-hash: 7df69f9469
  37   │       name: grafana-operated-deployment
  38   │     spec:
  39   │       containers:
  40   │       - args:
  41   │         - -config=/etc/grafana/grafana.ini
  42   │         env:
  43   │         - name: PLUGINS_HASH
  44   │           valueFrom:
  45   │             configMapKeyRef:
  46   │               key: PLUGINS_HASH
  47   │               name: grafana-operated-plugins
  48   │               optional: true
  49   │         - name: CONFIG_HASH
  50   │           value: 630f874a409e3060114f948c5b42696acfef6708d5ad33d015e8c1f118d79adf
  51   │         - name: GF_INSTALL_PLUGINS
  52   │           value: marcusolsson-treemap-panel 2.0.1,digrich-bubblechart-panel 2.0.1,marcusolsson-json-datasource
  53   │             1.3.9
  54   │         - name: TMPDIR
  55   │           value: /var/lib/grafana
  56   │         - name: GF_SECURITY_ADMIN_USER
  57   │           valueFrom:
  58   │             secretKeyRef:
  59   │               key: GF_SECURITY_ADMIN_USER
  60   │               name: grafana-operated-admin-credentials
  61   │         - name: GF_SECURITY_ADMIN_PASSWORD
  62   │           valueFrom:
  63   │             secretKeyRef:
  64   │               key: GF_SECURITY_ADMIN_PASSWORD
  65   │               name: grafana-operated-admin-credentials
  66   │         envFrom:
  67   │         - secretRef:
  68   │             name: grafana-auth-credentials-operations-internal-use1-eks
  69   │         image: grafana/grafana-oss:11.1.4-ubuntu
  70   │         imagePullPolicy: IfNotPresent
  71   │         name: grafana
  72   │         ports:
  73   │         - containerPort: 3000
  74   │           name: grafana-http
  75   │           protocol: TCP
  76   │         readinessProbe:
  77   │           failureThreshold: 1
  78   │           httpGet:
  79   │             path: /api/health
  80   │             port: 3000
  81   │             scheme: HTTP
  82   │           initialDelaySeconds: 5
  83   │           periodSeconds: 10
  84   │           successThreshold: 1
  85   │           timeoutSeconds: 3
  86   │         resources:
  87   │           limits:
  88   │             memory: 1Gi
  89   │           requests:
  90   │             cpu: 500m
  91   │             memory: 1Gi
  92   │         securityContext:
  93   │           allowPrivilegeEscalation: false
  94   │           capabilities:
  95   │             drop:
  96   │             - ALL
  97   │           privileged: false
  98   │           readOnlyRootFilesystem: true
  99   │           runAsGroup: 10001
 100   │           runAsNonRoot: true
 101   │           runAsUser: 10001
 102   │         terminationMessagePath: /dev/termination-log
 103   │         terminationMessagePolicy: File
 104   │         volumeMounts:
 105   │         - mountPath: /etc/grafana/
 106   │           name: grafana-operated-ini
 107   │         - mountPath: /var/lib/grafana
 108   │           name: grafana-data
 109   │         - mountPath: /var/log/grafana
 110   │           name: grafana-logs
 111   │       dnsPolicy: ClusterFirst
 112   │       nodeSelector:
 113   │         scope: monitoring
 114   │       restartPolicy: Always
 115   │       schedulerName: default-scheduler
 116   │       securityContext:
 117   │         fsGroup: 472
 118   │         seccompProfile:
 119   │           type: RuntimeDefault
 120   │         supplementalGroups:
 121   │         - 0
 122   │       serviceAccount: grafana-operated-sa
 123   │       serviceAccountName: grafana-operated-sa
 124   │       terminationGracePeriodSeconds: 30
 125   │       tolerations:
 126   │       - effect: NoSchedule
 127   │         key: Monitoring
 128   │         operator: Equal
 129   │         value: "true"
 130   │       volumes:
 131   │       - configMap:
 132   │           defaultMode: 420
 133   │           name: grafana-operated-ini
 134   │         name: grafana-operated-ini
 135   │       - emptyDir: {}
 136   │         name: grafana-logs
 137   │       - name: grafana-data
 138   │         persistentVolumeClaim:
 139   │           claimName: grafana-operated-pvc
 140   │ status:
 141   │   availableReplicas: 1
 142   │   fullyLabeledReplicas: 1
 143   │   observedGeneration: 1575
 144   │   readyReplicas: 1
 145   │   replicas: 1
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: grafana-operated-deployment-779fc75d95.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ apiVersion: apps/v1
   2   │ kind: ReplicaSet
   3   │ metadata:
   4   │   annotations:
   5   │     deployment.kubernetes.io/desired-replicas: "1"
   6   │     deployment.kubernetes.io/max-replicas: "1"
   7   │     deployment.kubernetes.io/revision: "6059"
   8   │     deployment.kubernetes.io/revision-history: 3556,3563,3565,3569,3575,3577,3585,3596,3600,3604,3612,3627,3630,3634,3638,3643,3646,3651,3661,3664,3667,3669,3672,3675,3693,3696,3698,370
       │ 9,3713,3720,3728,3730,3734,3737,3739,3741,3744,3747,3751,3763,3793,3805,3813,3828,3833,3837,3843,3845,3864,3870,3872,3876,3885,3889,3898,3908,3948,3961,3965,3967,3969,3981,3987,3989,399
       │ 1,4001,4004,4018,4028,4038,4048,4050,4065,4067,4069,4077,4081,4085,4090,4092,4118,4120,4125,4130,4142,4150,4153,4155,4159,4174,4180,4185,4187,4211,4215,4217,4237,4241,4244,4246,4253,425
       │ 6,4258,4260,4264,4270,4272,4274,4276,4279,4293,4295,4299,4313,4317,4321,4325,4327,4333,4339,4364,4367,4375,4391,4394,4399,4401,4408,4412,4414,4416,4426,4429,4434,4438,4441,4445,4451,445
       │ 5,4457,4460,4463,4466,4470,4473,4478,4480,4485,4494,4501,4512,4518,4523,4526,4528,4533,4535,4539,4543,4547,4552,4576,4580,4587,4595,4601,4605,4622,4626,4631,4642,4644,4648,4652,4663,466
       │ 5,4674,4676,4679,4690,4698,4720,4724,4726,4730,4733,4735,4742,4757,4768,4771,4779,4782,4786,4790,4795,4800,4818,4821,4823,4826,4836,4839,4843,4848,4850,4852,4859,4865,4870,4872,4874,489
       │ 5,4898,4901,4911,4914,4916,4921,4926,4933,4937,4939,4942,4955,4964,4972,4983,4986,4992,5006,5008,5012,5016,5018,5020,5028,5035,5038,5040,5043,5063,5066,5073,5080,5082,5084,5086,5088,509
       │ 2,5106,5112,5118,5120,5123,5125,5127,5135,5141,5143,5152,5155,5161,5165,5173,5179,5182,5194,5211,5215,5220,5223,5230,5239,5244,5249,5262,5266,5268,5273,5298,5300,5303,5307,5310,5316,532
       │ 2,5324,5328,5331,5335,5338,5340,5342,5357,5373,5380,5388,5393,5404,5417,5419,5430,5455,5461,5463,5467,5471,5476,5485,5487,5489,5494,5502,5512,5516,5524,5533,5538,5543,5553,5555,5558,556
       │ 0,5571,5573,5576,5588,5595,5608,5611,5622,5648,5651,5656,5660,5663,5675,5677,5679,5685,5691,5696,5705,5707,5709,5711,5715,5718,5727,5731,5739,5752,5756,5760,5765,5775,5778,5783,5791,579
       │ 5,5797,5800,5802,5806,5812,5815,5826,5846,5849,5862,5868,5870,5877,5888,5893,5898,5900,5911,5916,5929,5931,5940,5942,5944,5948,5950,5958,5978,5984,5991,6002,6005,6010,6012,6016,6020,602
       │ 2,6037,6052
   9   │   creationTimestamp: "2024-08-27T08:57:22Z"
  10   │   generation: 820
  11   │   labels:
  12   │     app: grafana-operated
  13   │     pod-template-hash: 779fc75d95
  14   │   name: grafana-operated-deployment-779fc75d95
  15   │   namespace: monitoring
  16   │   ownerReferences:
  17   │   - apiVersion: apps/v1
  18   │     blockOwnerDeletion: true
  19   │     controller: true
  20   │     kind: Deployment
  21   │     name: grafana-operated-deployment
  22   │     uid: d7edab22-c6f6-4c63-8fd4-d2fa8a0b798f
  23   │   resourceVersion: "633406527"
  24   │   uid: a553c0f3-12ac-4e43-8bc1-981090d9c433
  25   │ spec:
  26   │   replicas: 0
  27   │   selector:
  28   │     matchLabels:
  29   │       app: grafana-operated
  30   │       pod-template-hash: 779fc75d95
  31   │   template:
  32   │     metadata:
  33   │       creationTimestamp: null
  34   │       labels:
  35   │         app: grafana-operated
  36   │         pod-template-hash: 779fc75d95
  37   │       name: grafana-operated-deployment
  38   │     spec:
  39   │       containers:
  40   │       - args:
  41   │         - -config=/etc/grafana/grafana.ini
  42   │         env:
  43   │         - name: PLUGINS_HASH
  44   │           valueFrom:
  45   │             configMapKeyRef:
  46   │               key: PLUGINS_HASH
  47   │               name: grafana-operated-plugins
  48   │               optional: true
  49   │         - name: CONFIG_HASH
  50   │           value: 630f874a409e3060114f948c5b42696acfef6708d5ad33d015e8c1f118d79adf
  51   │         - name: GF_INSTALL_PLUGINS
  52   │           value: digrich-bubblechart-panel 2.0.1,marcusolsson-treemap-panel 2.0.1,marcusolsson-json-datasource
  53   │             1.3.9
  54   │         - name: TMPDIR
  55   │           value: /var/lib/grafana
  56   │         - name: GF_SECURITY_ADMIN_USER
  57   │           valueFrom:
  58   │             secretKeyRef:
  59   │               key: GF_SECURITY_ADMIN_USER
  60   │               name: grafana-operated-admin-credentials
  61   │         - name: GF_SECURITY_ADMIN_PASSWORD
  62   │           valueFrom:
  63   │             secretKeyRef:
  64   │               key: GF_SECURITY_ADMIN_PASSWORD
  65   │               name: grafana-operated-admin-credentials
  66   │         envFrom:
  67   │         - secretRef:
  68   │             name: grafana-auth-credentials-operations-internal-use1-eks
  69   │         image: grafana/grafana-oss:11.1.4-ubuntu
  70   │         imagePullPolicy: IfNotPresent
  71   │         name: grafana
  72   │         ports:
  73   │         - containerPort: 3000
  74   │           name: grafana-http
  75   │           protocol: TCP
  76   │         readinessProbe:
  77   │           failureThreshold: 1
  78   │           httpGet:
  79   │             path: /api/health
  80   │             port: 3000
  81   │             scheme: HTTP
  82   │           initialDelaySeconds: 5
  83   │           periodSeconds: 10
  84   │           successThreshold: 1
  85   │           timeoutSeconds: 3
  86   │         resources:
  87   │           limits:
  88   │             memory: 1Gi
  89   │           requests:
  90   │             cpu: 500m
  91   │             memory: 1Gi
  92   │         securityContext:
  93   │           allowPrivilegeEscalation: false
  94   │           capabilities:
  95   │             drop:
  96   │             - ALL
  97   │           privileged: false
  98   │           readOnlyRootFilesystem: true
  99   │           runAsGroup: 10001
 100   │           runAsNonRoot: true
 101   │           runAsUser: 10001
 102   │         terminationMessagePath: /dev/termination-log
 103   │         terminationMessagePolicy: File
 104   │         volumeMounts:
 105   │         - mountPath: /etc/grafana/
 106   │           name: grafana-operated-ini
 107   │         - mountPath: /var/lib/grafana
 108   │           name: grafana-data
 109   │         - mountPath: /var/log/grafana
 110   │           name: grafana-logs
 111   │       dnsPolicy: ClusterFirst
 112   │       nodeSelector:
 113   │         scope: monitoring
 114   │       restartPolicy: Always
 115   │       schedulerName: default-scheduler
 116   │       securityContext:
 117   │         fsGroup: 472
 118   │         seccompProfile:
 119   │           type: RuntimeDefault
 120   │         supplementalGroups:
 121   │         - 0
 122   │       serviceAccount: grafana-operated-sa
 123   │       serviceAccountName: grafana-operated-sa
 124   │       terminationGracePeriodSeconds: 30
 125   │       tolerations:
 126   │       - effect: NoSchedule
 127   │         key: Monitoring
 128   │         operator: Equal
 129   │         value: "true"
 130   │       volumes:
 131   │       - configMap:
 132   │           defaultMode: 420
 133   │           name: grafana-operated-ini
 134   │         name: grafana-operated-ini
 135   │       - emptyDir: {}
 136   │         name: grafana-logs
 137   │       - name: grafana-data
 138   │         persistentVolumeClaim:
 139   │           claimName: grafana-operated-pvc
 140   │ status:
 141   │   observedGeneration: 820
 142   │   replicas: 0
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: grafana-operated-deployment-858d55865c.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ apiVersion: apps/v1
   2   │ kind: ReplicaSet
   3   │ metadata:
   4   │   annotations:
   5   │     deployment.kubernetes.io/desired-replicas: "1"
   6   │     deployment.kubernetes.io/max-replicas: "1"
   7   │     deployment.kubernetes.io/revision: "6057"
   8   │     deployment.kubernetes.io/revision-history: 3732,3736,3740,3750,3753,3758,3760,3766,3769,3773,3775,3779,3781,3789,3792,3798,3800,3816,3819,3826,3830,3838,3842,3848,3852,3854,3857,385
       │ 9,3865,3869,3884,3886,3893,3899,3906,3912,3914,3920,3922,3925,3931,3933,3936,3939,3941,3947,3954,3956,3963,3966,3968,3976,3983,4007,4011,4013,4026,4029,4036,4041,4044,4054,4056,4060,407
       │ 6,4079,4094,4097,4099,4102,4108,4110,4123,4127,4131,4133,4138,4140,4146,4149,4157,4160,4164,4166,4168,4172,4175,4179,4191,4195,4198,4201,4204,4206,4210,4229,4231,4234,4249,4261,4273,427
       │ 7,4283,4288,4302,4305,4308,4324,4330,4335,4340,4346,4348,4353,4356,4361,4365,4368,4372,4376,4382,4387,4389,4393,4396,4405,4409,4413,4419,4422,4424,4428,4444,4447,4453,4458,4462,4469,447
       │ 1,4486,4490,4496,4505,4507,4525,4527,4532,4542,4545,4550,4555,4557,4570,4573,4575,4578,4583,4585,4591,4607,4609,4620,4633,4636,4640,4651,4655,4660,4667,4672,4689,4694,4697,4699,4706,470
       │ 8,4713,4721,4737,4740,4746,4749,4752,4755,4764,4770,4775,4777,4789,4793,4796,4804,4815,4822,4829,4831,4838,4840,4846,4849,4855,4862,4867,4869,4875,4877,4883,4886,4900,4902,4907,4920,492
       │ 3,4925,4929,4946,4950,4968,4971,4974,4977,4979,4981,4987,4990,4994,4996,4998,5001,5004,5009,5014,5022,5030,5032,5037,5048,5054,5056,5060,5064,5068,5071,5095,5099,5103,5110,5121,5132,513
       │ 6,5138,5149,5151,5154,5157,5162,5169,5171,5175,5186,5189,5192,5205,5209,5213,5216,5219,5232,5234,5240,5250,5256,5258,5264,5270,5277,5284,5286,5293,5295,5305,5309,5312,5315,5319,5321,533
       │ 0,5332,5334,5336,5347,5350,5352,5355,5359,5364,5367,5369,5378,5384,5386,5424,5437,5446,5451,5459,5472,5475,5490,5492,5495,5498,5500,5513,5531,5535,5540,5544,5546,5548,5561,5564,5566,556
       │ 9,5575,5581,5585,5593,5607,5616,5618,5628,5630,5634,5638,5640,5643,5646,5654,5657,5666,5673,5682,5689,5693,5700,5704,5708,5726,5740,5745,5755,5762,5781,5785,5787,5793,5798,5801,5804,580
       │ 7,5822,5825,5828,5833,5838,5842,5859,5863,5866,5882,5884,5896,5899,5902,5905,5907,5913,5917,5919,5926,5941,5943,5953,5956,5959,5961,5965,5973,5975,5994,6000,6013,6025,6030,6032,6036,604
       │ 6,6048,6053
   9   │   creationTimestamp: "2024-08-27T08:57:25Z"
  10   │   generation: 923
  11   │   labels:
  12   │     app: grafana-operated
  13   │     pod-template-hash: 858d55865c
  14   │   name: grafana-operated-deployment-858d55865c
  15   │   namespace: monitoring
  16   │   ownerReferences:
  17   │   - apiVersion: apps/v1
  18   │     blockOwnerDeletion: true
  19   │     controller: true
  20   │     kind: Deployment
  21   │     name: grafana-operated-deployment
  22   │     uid: d7edab22-c6f6-4c63-8fd4-d2fa8a0b798f
  23   │   resourceVersion: "633406180"
  24   │   uid: 8dd6c75f-e134-418b-bc38-57454d2c432b
  25   │ spec:
  26   │   replicas: 0
  27   │   selector:
  28   │     matchLabels:
  29   │       app: grafana-operated
  30   │       pod-template-hash: 858d55865c
  31   │   template:
  32   │     metadata:
  33   │       creationTimestamp: null
  34   │       labels:
  35   │         app: grafana-operated
  36   │         pod-template-hash: 858d55865c
  37   │       name: grafana-operated-deployment
  38   │     spec:
  39   │       containers:
  40   │       - args:
  41   │         - -config=/etc/grafana/grafana.ini
  42   │         env:
  43   │         - name: PLUGINS_HASH
  44   │           valueFrom:
  45   │             configMapKeyRef:
  46   │               key: PLUGINS_HASH
  47   │               name: grafana-operated-plugins
  48   │               optional: true
  49   │         - name: CONFIG_HASH
  50   │           value: 630f874a409e3060114f948c5b42696acfef6708d5ad33d015e8c1f118d79adf
  51   │         - name: GF_INSTALL_PLUGINS
  52   │           value: marcusolsson-json-datasource 1.3.9,marcusolsson-treemap-panel 2.0.1,digrich-bubblechart-panel
  53   │             2.0.1
  54   │         - name: TMPDIR
  55   │           value: /var/lib/grafana
  56   │         - name: GF_SECURITY_ADMIN_USER
  57   │           valueFrom:
  58   │             secretKeyRef:
  59   │               key: GF_SECURITY_ADMIN_USER
  60   │               name: grafana-operated-admin-credentials
  61   │         - name: GF_SECURITY_ADMIN_PASSWORD
  62   │           valueFrom:
  63   │             secretKeyRef:
  64   │               key: GF_SECURITY_ADMIN_PASSWORD
  65   │               name: grafana-operated-admin-credentials
  66   │         envFrom:
  67   │         - secretRef:
  68   │             name: grafana-auth-credentials-operations-internal-use1-eks
  69   │         image: grafana/grafana-oss:11.1.4-ubuntu
  70   │         imagePullPolicy: IfNotPresent
  71   │         name: grafana
  72   │         ports:
  73   │         - containerPort: 3000
  74   │           name: grafana-http
  75   │           protocol: TCP
  76   │         readinessProbe:
  77   │           failureThreshold: 1
  78   │           httpGet:
  79   │             path: /api/health
  80   │             port: 3000
  81   │             scheme: HTTP
  82   │           initialDelaySeconds: 5
  83   │           periodSeconds: 10
  84   │           successThreshold: 1
  85   │           timeoutSeconds: 3
  86   │         resources:
  87   │           limits:
  88   │             memory: 1Gi
  89   │           requests:
  90   │             cpu: 500m
  91   │             memory: 1Gi
  92   │         securityContext:
  93   │           allowPrivilegeEscalation: false
  94   │           capabilities:
  95   │             drop:
  96   │             - ALL
  97   │           privileged: false
  98   │           readOnlyRootFilesystem: true
  99   │           runAsGroup: 10001
 100   │           runAsNonRoot: true
 101   │           runAsUser: 10001
 102   │         terminationMessagePath: /dev/termination-log
 103   │         terminationMessagePolicy: File
 104   │         volumeMounts:
 105   │         - mountPath: /etc/grafana/
 106   │           name: grafana-operated-ini
 107   │         - mountPath: /var/lib/grafana
 108   │           name: grafana-data
 109   │         - mountPath: /var/log/grafana
 110   │           name: grafana-logs
 111   │       dnsPolicy: ClusterFirst
 112   │       nodeSelector:
 113   │         scope: monitoring
 114   │       restartPolicy: Always
 115   │       schedulerName: default-scheduler
 116   │       securityContext:
 117   │         fsGroup: 472
 118   │         seccompProfile:
 119   │           type: RuntimeDefault
 120   │         supplementalGroups:
 121   │         - 0
 122   │       serviceAccount: grafana-operated-sa
 123   │       serviceAccountName: grafana-operated-sa
 124   │       terminationGracePeriodSeconds: 30
 125   │       tolerations:
 126   │       - effect: NoSchedule
 127   │         key: Monitoring
 128   │         operator: Equal
 129   │         value: "true"
 130   │       volumes:
 131   │       - configMap:
 132   │           defaultMode: 420
 133   │           name: grafana-operated-ini
 134   │         name: grafana-operated-ini
 135   │       - emptyDir: {}
 136   │         name: grafana-logs
 137   │       - name: grafana-data
 138   │         persistentVolumeClaim:
 139   │           claimName: grafana-operated-pvc
 140   │ status:
 141   │   observedGeneration: 923
 142   │   replicas: 0
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: grafana-operated-deployment-78488687f.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ apiVersion: apps/v1
   2   │ kind: ReplicaSet
   3   │ metadata:
   4   │   annotations:
   5   │     deployment.kubernetes.io/desired-replicas: "1"
   6   │     deployment.kubernetes.io/max-replicas: "1"
   7   │     deployment.kubernetes.io/revision: "6060"
   8   │     deployment.kubernetes.io/revision-history: 4598,4600,4604,4611,4614,4618,4621,4623,4625,4630,4632,4638,4643,4646,4649,4653,4656,4659,4662,4664,4669,4675,4678,4681,4686,4688,4691,469
       │ 3,4696,4700,4702,4704,4709,4712,4715,4718,4723,4728,4731,4734,4739,4741,4743,4745,4747,4751,4754,4756,4759,4761,4765,4773,4776,4778,4780,4783,4785,4787,4792,4794,4797,4802,4808,4810,481
       │ 2,4814,4816,4819,4825,4828,4830,4834,4837,4842,4844,4847,4851,4853,4860,4864,4866,4871,4876,4882,4884,4891,4896,4903,4905,4908,4913,4915,4917,4919,4927,4930,4932,4935,4941,4945,4948,495
       │ 2,4954,4957,4960,4962,4966,4969,4976,4980,4985,4988,4993,4995,5000,5003,5010,5015,5019,5023,5025,5027,5033,5039,5042,5045,5047,5052,5055,5057,5059,5062,5067,5069,5076,5078,5081,5083,508
       │ 9,5094,5097,5100,5104,5107,5109,5115,5122,5126,5128,5130,5134,5139,5142,5145,5153,5160,5164,5167,5170,5177,5181,5183,5185,5187,5191,5193,5195,5197,5199,5201,5203,5206,5208,5212,5217,522
       │ 4,5228,5233,5235,5237,5241,5243,5245,5247,5252,5255,5259,5261,5263,5272,5275,5278,5280,5283,5288,5290,5292,5296,5301,5304,5306,5308,5313,5326,5341,5344,5346,5349,5354,5356,5358,5362,536
       │ 5,5372,5374,5376,5379,5383,5387,5389,5391,5395,5398,5400,5403,5406,5409,5411,5414,5418,5422,5426,5428,5432,5434,5436,5440,5454,5457,5460,5464,5466,5469,5477,5479,5481,5483,5493,5496,550
       │ 1,5503,5505,5509,5511,5515,5517,5519,5522,5529,5532,5536,5542,5545,5547,5552,5554,5556,5568,5570,5572,5577,5580,5587,5597,5599,5601,5603,5605,5609,5613,5615,5617,5619,5621,5624,5626,562
       │ 9,5631,5635,5637,5645,5649,5653,5659,5665,5668,5672,5678,5681,5684,5687,5694,5697,5699,5702,5713,5721,5723,5725,5728,5730,5733,5738,5742,5744,5747,5749,5751,5757,5763,5767,5770,5772,577
       │ 7,5780,5786,5789,5792,5803,5805,5808,5810,5813,5816,5818,5820,5823,5829,5831,5840,5843,5845,5847,5851,5853,5857,5865,5872,5874,5876,5878,5880,5886,5890,5892,5894,5897,5901,5903,5909,591
       │ 4,5918,5924,5927,5933,5935,5937,5939,5945,5951,5955,5960,5963,5969,5971,5977,5979,5981,5985,5987,5989,5993,5996,5998,6001,6004,6006,6009,6021,6023,6027,6029,6031,6033,6035,6041,6044,605
       │ 0,6055,6058
   9   │   creationTimestamp: "2024-08-27T08:57:22Z"
  10   │   generation: 1620
  11   │   labels:
  12   │     app: grafana-operated
  13   │     pod-template-hash: 78488687f
  14   │   name: grafana-operated-deployment-78488687f
  15   │   namespace: monitoring
  16   │   ownerReferences:
  17   │   - apiVersion: apps/v1
  18   │     blockOwnerDeletion: true
  19   │     controller: true
  20   │     kind: Deployment
  21   │     name: grafana-operated-deployment
  22   │     uid: d7edab22-c6f6-4c63-8fd4-d2fa8a0b798f
  23   │   resourceVersion: "633406753"
  24   │   uid: 8ca50ebd-b163-4279-9c03-952f364ff46e
  25   │ spec:
  26   │   replicas: 0
  27   │   selector:
  28   │     matchLabels:
  29   │       app: grafana-operated
  30   │       pod-template-hash: 78488687f
  31   │   template:
  32   │     metadata:
  33   │       creationTimestamp: null
  34   │       labels:
  35   │         app: grafana-operated
  36   │         pod-template-hash: 78488687f
  37   │       name: grafana-operated-deployment
  38   │     spec:
  39   │       containers:
  40   │       - args:
  41   │         - -config=/etc/grafana/grafana.ini
  42   │         env:
  43   │         - name: PLUGINS_HASH
  44   │           valueFrom:
  45   │             configMapKeyRef:
  46   │               key: PLUGINS_HASH
  47   │               name: grafana-operated-plugins
  48   │               optional: true
  49   │         - name: CONFIG_HASH
  50   │           value: 630f874a409e3060114f948c5b42696acfef6708d5ad33d015e8c1f118d79adf
  51   │         - name: GF_INSTALL_PLUGINS
  52   │           value: marcusolsson-treemap-panel 2.0.1,marcusolsson-json-datasource 1.3.9,digrich-bubblechart-panel
  53   │             2.0.1
  54   │         - name: TMPDIR
  55   │           value: /var/lib/grafana
  56   │         - name: GF_SECURITY_ADMIN_USER
  57   │           valueFrom:
  58   │             secretKeyRef:
  59   │               key: GF_SECURITY_ADMIN_USER
  60   │               name: grafana-operated-admin-credentials
  61   │         - name: GF_SECURITY_ADMIN_PASSWORD
  62   │           valueFrom:
  63   │             secretKeyRef:
  64   │               key: GF_SECURITY_ADMIN_PASSWORD
  65   │               name: grafana-operated-admin-credentials
  66   │         envFrom:
  67   │         - secretRef:
  68   │             name: grafana-auth-credentials-operations-internal-use1-eks
  69   │         image: grafana/grafana-oss:11.1.4-ubuntu
  70   │         imagePullPolicy: IfNotPresent
  71   │         name: grafana
  72   │         ports:
  73   │         - containerPort: 3000
  74   │           name: grafana-http
  75   │           protocol: TCP
  76   │         readinessProbe:
  77   │           failureThreshold: 1
  78   │           httpGet:
  79   │             path: /api/health
  80   │             port: 3000
  81   │             scheme: HTTP
  82   │           initialDelaySeconds: 5
  83   │           periodSeconds: 10
  84   │           successThreshold: 1
  85   │           timeoutSeconds: 3
  86   │         resources:
  87   │           limits:
  88   │             memory: 1Gi
  89   │           requests:
  90   │             cpu: 500m
  91   │             memory: 1Gi
  92   │         securityContext:
  93   │           allowPrivilegeEscalation: false
  94   │           capabilities:
  95   │             drop:
  96   │             - ALL
  97   │           privileged: false
  98   │           readOnlyRootFilesystem: true
  99   │           runAsGroup: 10001
 100   │           runAsNonRoot: true
 101   │           runAsUser: 10001
 102   │         terminationMessagePath: /dev/termination-log
 103   │         terminationMessagePolicy: File
 104   │         volumeMounts:
 105   │         - mountPath: /etc/grafana/
 106   │           name: grafana-operated-ini
 107   │         - mountPath: /var/lib/grafana
 108   │           name: grafana-data
 109   │         - mountPath: /var/log/grafana
 110   │           name: grafana-logs
 111   │       dnsPolicy: ClusterFirst
 112   │       nodeSelector:
 113   │         scope: monitoring
 114   │       restartPolicy: Always
 115   │       schedulerName: default-scheduler
 116   │       securityContext:
 117   │         fsGroup: 472
 118   │         seccompProfile:
 119   │           type: RuntimeDefault
 120   │         supplementalGroups:
 121   │         - 0
 122   │       serviceAccount: grafana-operated-sa
 123   │       serviceAccountName: grafana-operated-sa
 124   │       terminationGracePeriodSeconds: 30
 125   │       tolerations:
 126   │       - effect: NoSchedule
 127   │         key: Monitoring
 128   │         operator: Equal
 129   │         value: "true"
 130   │       volumes:
 131   │       - configMap:
 132   │           defaultMode: 420
 133   │           name: grafana-operated-ini
 134   │         name: grafana-operated-ini
 135   │       - emptyDir: {}
 136   │         name: grafana-logs
 137   │       - name: grafana-data
 138   │         persistentVolumeClaim:
 139   │           claimName: grafana-operated-pvc
 140   │ status:
 141   │   observedGeneration: 1620
 142   │   replicas: 0
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: grafana-operated-deployment-c69d6557c.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ apiVersion: apps/v1
   2   │ kind: ReplicaSet
   3   │ metadata:
   4   │   annotations:
   5   │     deployment.kubernetes.io/desired-replicas: "1"
   6   │     deployment.kubernetes.io/max-replicas: "1"
   7   │     deployment.kubernetes.io/revision: "6043"
   8   │     deployment.kubernetes.io/revision-history: 22,42,44,56,63,71,77,110,151,166,180,194,222,230,237,250,252,261,277,281,319,327,335,349,366,382,396,414,426,465,467,515,531,534,542,594,5
       │ 98,609,629,637,640,651,669,688,694,696,712,735,747,859,893,920,926,948,958,961,965,968,986,1005,1014,1053,1071,1082,1100,1129,1131,1133,1173,1184,1193,1198,1201,1205,1208,1232,1234,1238
       │ ,1253,1261,1270,1275,1317,1325,1337,1350,1370,1379,1381,1390,1394,1408,1423,1437,1441,1445,1463,1471,1487,1497,1508,1516,1524,1561,1565,1603,1607,1613,1636,1647,1652,1656,1682,1700,1710
       │ ,1744,1747,1752,1763,1826,1868,1879,1908,1914,1929,1944,1981,1997,2018,2022,2035,2044,2085,2090,2104,2106,2132,2139,2174,2192,2218,2232,2245,2250,2275,2278,2284,2286,2301,2310,2325,2343
       │ ,2345,2347,2353,2357,2364,2373,2379,2391,2399,2403,2405,2424,2440,2460,2469,2519,2527,2530,2532,2535,2540,2579,2589,2615,2646,2654,2697,2716,2728,2736,2738,2763,2820,2830,2837,2847,2864
       │ ,2868,2887,2905,2919,2924,2939,2943,3002,3006,3047,3076,3094,3096,3099,3119,3139,3204,3213,3215,3229,3251,3268,3277,3279,3316,3330,3343,3376,3418,3446,3468,3471,3474,3482,3512,3526,3540
       │ ,3547,3571,3579,3581,3583,3588,3597,3599,3609,3615,3619,3641,3659,3691,3716,3726,3748,3771,3777,3802,3808,3834,3850,3862,3868,3879,3892,3901,3903,3905,3930,3960,3971,3998,4014,4019,4033
       │ ,4051,4059,4074,4086,4106,4122,4136,4177,4183,4218,4221,4242,4247,4250,4255,4268,4275,4306,4319,4328,4402,4510,4541,4544,4551,4561,4564,4568,4592,4613,4616,4635,4680,4682,4684,4705,4717
       │ ,4722,4732,4762,4766,4769,4798,4803,4806,4813,4820,4832,4857,4873,4889,4893,4963,4967,5050,5077,5091,5096,5098,5105,5111,5131,5133,5140,5147,5204,5207,5222,5226,5281,5287,5291,5294,5318
       │ ,5345,5360,5397,5401,5408,5438,5443,5445,5448,5488,5507,5526,5550,5559,5562,5567,5578,5589,5600,5602,5606,5620,5641,5670,5698,5710,5714,5720,5722,5735,5768,5774,5827,5855,5864,5871,5875
       │ ,5879,5891,5921,5925,5930,5967,6015,6028,6039
   9   │   creationTimestamp: "2024-08-27T08:58:01Z"
  10   │   generation: 279
  11   │   labels:
  12   │     app: grafana-operated
  13   │     pod-template-hash: c69d6557c
  14   │   name: grafana-operated-deployment-c69d6557c
  15   │   namespace: monitoring
  16   │   ownerReferences:
  17   │   - apiVersion: apps/v1
  18   │     blockOwnerDeletion: true
  19   │     controller: true
  20   │     kind: Deployment
  21   │     name: grafana-operated-deployment
  22   │     uid: d7edab22-c6f6-4c63-8fd4-d2fa8a0b798f
  23   │   resourceVersion: "633404411"
  24   │   uid: 6aa30210-341f-42e8-8608-306286412843
  25   │ spec:
  26   │   replicas: 0
  27   │   selector:
  28   │     matchLabels:
  29   │       app: grafana-operated
  30   │       pod-template-hash: c69d6557c
  31   │   template:
  32   │     metadata:
  33   │       creationTimestamp: null
  34   │       labels:
  35   │         app: grafana-operated
  36   │         pod-template-hash: c69d6557c
  37   │       name: grafana-operated-deployment
  38   │     spec:
  39   │       containers:
  40   │       - args:
  41   │         - -config=/etc/grafana/grafana.ini
  42   │         env:
  43   │         - name: PLUGINS_HASH
  44   │           valueFrom:
  45   │             configMapKeyRef:
  46   │               key: PLUGINS_HASH
  47   │               name: grafana-operated-plugins
  48   │               optional: true
  49   │         - name: CONFIG_HASH
  50   │           value: 630f874a409e3060114f948c5b42696acfef6708d5ad33d015e8c1f118d79adf
  51   │         - name: GF_INSTALL_PLUGINS
  52   │           value: digrich-bubblechart-panel 2.0.1,marcusolsson-json-datasource 1.3.9,marcusolsson-treemap-panel
  53   │             2.0.1
  54   │         - name: TMPDIR
  55   │           value: /var/lib/grafana
  56   │         - name: GF_SECURITY_ADMIN_USER
  57   │           valueFrom:
  58   │             secretKeyRef:
  59   │               key: GF_SECURITY_ADMIN_USER
  60   │               name: grafana-operated-admin-credentials
  61   │         - name: GF_SECURITY_ADMIN_PASSWORD
  62   │           valueFrom:
  63   │             secretKeyRef:
  64   │               key: GF_SECURITY_ADMIN_PASSWORD
  65   │               name: grafana-operated-admin-credentials
  66   │         envFrom:
  67   │         - secretRef:
  68   │             name: grafana-auth-credentials-operations-internal-use1-eks
  69   │         image: grafana/grafana-oss:11.1.4-ubuntu
  70   │         imagePullPolicy: IfNotPresent
  71   │         name: grafana
  72   │         ports:
  73   │         - containerPort: 3000
  74   │           name: grafana-http
  75   │           protocol: TCP
  76   │         readinessProbe:
  77   │           failureThreshold: 1
  78   │           httpGet:
  79   │             path: /api/health
  80   │             port: 3000
  81   │             scheme: HTTP
  82   │           initialDelaySeconds: 5
  83   │           periodSeconds: 10
  84   │           successThreshold: 1
  85   │           timeoutSeconds: 3
  86   │         resources:
  87   │           limits:
  88   │             memory: 1Gi
  89   │           requests:
  90   │             cpu: 500m
  91   │             memory: 1Gi
  92   │         securityContext:
  93   │           allowPrivilegeEscalation: false
  94   │           capabilities:
  95   │             drop:
  96   │             - ALL
  97   │           privileged: false
  98   │           readOnlyRootFilesystem: true
  99   │           runAsGroup: 10001
 100   │           runAsNonRoot: true
 101   │           runAsUser: 10001
 102   │         terminationMessagePath: /dev/termination-log
 103   │         terminationMessagePolicy: File
 104   │         volumeMounts:
 105   │         - mountPath: /etc/grafana/
 106   │           name: grafana-operated-ini
 107   │         - mountPath: /var/lib/grafana
 108   │           name: grafana-data
 109   │         - mountPath: /var/log/grafana
 110   │           name: grafana-logs
 111   │       dnsPolicy: ClusterFirst
 112   │       nodeSelector:
 113   │         scope: monitoring
 114   │       restartPolicy: Always
 115   │       schedulerName: default-scheduler
 116   │       securityContext:
 117   │         fsGroup: 472
 118   │         seccompProfile:
 119   │           type: RuntimeDefault
 120   │         supplementalGroups:
 121   │         - 0
 122   │       serviceAccount: grafana-operated-sa
 123   │       serviceAccountName: grafana-operated-sa
 124   │       terminationGracePeriodSeconds: 30
 125   │       tolerations:
 126   │       - effect: NoSchedule
 127   │         key: Monitoring
 128   │         operator: Equal
 129   │         value: "true"
 130   │       volumes:
 131   │       - configMap:
 132   │           defaultMode: 420
 133   │           name: grafana-operated-ini
 134   │         name: grafana-operated-ini
 135   │       - emptyDir: {}
 136   │         name: grafana-logs
 137   │       - name: grafana-data
 138   │         persistentVolumeClaim:
 139   │           claimName: grafana-operated-pvc
 140   │ status:
 141   │   observedGeneration: 279
 142   │   replicas: 0
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: grafana-operated-deployment-fb6c9ff47.yaml
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ apiVersion: apps/v1
   2   │ kind: ReplicaSet
   3   │ metadata:
   4   │   annotations:
   5   │     deployment.kubernetes.io/desired-replicas: "1"
   6   │     deployment.kubernetes.io/max-replicas: "1"
   7   │     deployment.kubernetes.io/revision: "6056"
   8   │     deployment.kubernetes.io/revision-history: 6,12,39,49,51,84,97,117,119,148,156,158,163,189,191,212,214,220,269,285,362,367,374,378,423,470,494,496,520,524,560,578,608,613,631,675,67
       │ 7,683,687,705,725,746,750,752,764,769,793,837,839,852,885,888,917,940,945,950,960,974,1025,1039,1049,1060,1106,1114,1136,1213,1221,1226,1231,1250,1255,1262,1281,1298,1308,1330,1366,1374
       │ ,1376,1411,1426,1456,1460,1464,1549,1569,1571,1609,1624,1628,1654,1680,1688,1713,1716,1743,1780,1801,1803,1808,1814,1825,1834,1843,1860,1869,1875,1881,1902,1909,1934,1937,1959,1993,2023
       │ ,2029,2042,2115,2159,2187,2199,2201,2206,2212,2234,2251,2266,2280,2307,2314,2316,2330,2337,2352,2369,2377,2433,2453,2577,2587,2609,2613,2616,2645,2676,2718,2724,2729,2733,2770,2782,2784
       │ ,2816,2871,2888,2893,2897,2931,2934,2955,2963,2968,2988,2992,3008,3029,3032,3050,3057,3065,3080,3085,3103,3110,3164,3170,3191,3239,3248,3296,3328,3348,3374,3380,3387,3399,3404,3409,3423
       │ ,3427,3432,3435,3443,3450,3455,3479,3485,3488,3496,3501,3506,3521,3524,3545,3551,3586,3594,3602,3607,3617,3623,3635,3685,3705,3707,3710,3762,3778,3786,3796,3807,3810,3832,3847,3849,3856
       │ ,3867,3871,3874,3895,3900,3902,3915,3955,3959,3980,3984,4032,4046,4063,4080,4113,4154,4189,4205,4235,4252,4257,4281,4292,4294,4296,4304,4310,4312,4318,4331,4342,4350,4363,4384,4407,4468
       │ ,4483,4491,4506,4517,4519,4559,4566,4581,4597,4602,4627,4629,4634,4650,4658,4661,4666,4670,4677,4710,4736,4748,4750,4861,4878,4880,4887,4897,4910,4936,4938,4944,4959,4965,4999,5075,5102
       │ ,5113,5117,5119,5144,5159,5166,5180,5198,5214,5236,5254,5257,5269,5271,5297,5317,5327,5371,5381,5402,5413,5415,5420,5441,5450,5452,5465,5474,5521,5528,5551,5563,5579,5583,5591,5594,5596
       │ ,5625,5632,5661,5676,5706,5712,5716,5719,5737,5743,5753,5758,5766,5796,5835,5837,5856,5860,5869,5883,5908,5922,5932,5936,5947,5983,5986,6008,6018,6047
   9   │   creationTimestamp: "2024-08-27T08:57:25Z"
  10   │   generation: 299
  11   │   labels:
  12   │     app: grafana-operated
  13   │     pod-template-hash: fb6c9ff47
  14   │   name: grafana-operated-deployment-fb6c9ff47
  15   │   namespace: monitoring
  16   │   ownerReferences:
  17   │   - apiVersion: apps/v1
  18   │     blockOwnerDeletion: true
  19   │     controller: true
  20   │     kind: Deployment
  21   │     name: grafana-operated-deployment
  22   │     uid: d7edab22-c6f6-4c63-8fd4-d2fa8a0b798f
  23   │   resourceVersion: "633406187"
  24   │   uid: 381900ed-3d37-4c0a-aac2-84917af3575a
  25   │ spec:
  26   │   replicas: 0
  27   │   selector:
  28   │     matchLabels:
  29   │       app: grafana-operated
  30   │       pod-template-hash: fb6c9ff47
  31   │   template:
  32   │     metadata:
  33   │       creationTimestamp: null
  34   │       labels:
  35   │         app: grafana-operated
  36   │         pod-template-hash: fb6c9ff47
  37   │       name: grafana-operated-deployment
  38   │     spec:
  39   │       containers:
  40   │       - args:
  41   │         - -config=/etc/grafana/grafana.ini
  42   │         env:
  43   │         - name: PLUGINS_HASH
  44   │           valueFrom:
  45   │             configMapKeyRef:
  46   │               key: PLUGINS_HASH
  47   │               name: grafana-operated-plugins
  48   │               optional: true
  49   │         - name: CONFIG_HASH
  50   │           value: 630f874a409e3060114f948c5b42696acfef6708d5ad33d015e8c1f118d79adf
  51   │         - name: GF_INSTALL_PLUGINS
  52   │           value: marcusolsson-json-datasource 1.3.9,digrich-bubblechart-panel 2.0.1,marcusolsson-treemap-panel
  53   │             2.0.1
  54   │         - name: TMPDIR
  55   │           value: /var/lib/grafana
  56   │         - name: GF_SECURITY_ADMIN_USER
  57   │           valueFrom:
  58   │             secretKeyRef:
  59   │               key: GF_SECURITY_ADMIN_USER
  60   │               name: grafana-operated-admin-credentials
  61   │         - name: GF_SECURITY_ADMIN_PASSWORD
  62   │           valueFrom:
  63   │             secretKeyRef:
  64   │               key: GF_SECURITY_ADMIN_PASSWORD
  65   │               name: grafana-operated-admin-credentials
  66   │         envFrom:
  67   │         - secretRef:
  68   │             name: grafana-auth-credentials-operations-internal-use1-eks
  69   │         image: grafana/grafana-oss:11.1.4-ubuntu
  70   │         imagePullPolicy: IfNotPresent
  71   │         name: grafana
  72   │         ports:
  73   │         - containerPort: 3000
  74   │           name: grafana-http
  75   │           protocol: TCP
  76   │         readinessProbe:
  77   │           failureThreshold: 1
  78   │           httpGet:
  79   │             path: /api/health
  80   │             port: 3000
  81   │             scheme: HTTP
  82   │           initialDelaySeconds: 5
  83   │           periodSeconds: 10
  84   │           successThreshold: 1
  85   │           timeoutSeconds: 3
  86   │         resources:
  87   │           limits:
  88   │             memory: 1Gi
  89   │           requests:
  90   │             cpu: 500m
  91   │             memory: 1Gi
  92   │         securityContext:
  93   │           allowPrivilegeEscalation: false
  94   │           capabilities:
  95   │             drop:
  96   │             - ALL
  97   │           privileged: false
  98   │           readOnlyRootFilesystem: true
  99   │           runAsGroup: 10001
 100   │           runAsNonRoot: true
 101   │           runAsUser: 10001
 102   │         terminationMessagePath: /dev/termination-log
 103   │         terminationMessagePolicy: File
 104   │         volumeMounts:
 105   │         - mountPath: /etc/grafana/
 106   │           name: grafana-operated-ini
 107   │         - mountPath: /var/lib/grafana
 108   │           name: grafana-data
 109   │         - mountPath: /var/log/grafana
 110   │           name: grafana-logs
 111   │       dnsPolicy: ClusterFirst
 112   │       nodeSelector:
 113   │         scope: monitoring
 114   │       restartPolicy: Always
 115   │       schedulerName: default-scheduler
 116   │       securityContext:
 117   │         fsGroup: 472
 118   │         seccompProfile:
 119   │           type: RuntimeDefault
 120   │         supplementalGroups:
 121   │         - 0
 122   │       serviceAccount: grafana-operated-sa
 123   │       serviceAccountName: grafana-operated-sa
 124   │       terminationGracePeriodSeconds: 30
 125   │       tolerations:
 126   │       - effect: NoSchedule
 127   │         key: Monitoring
 128   │         operator: Equal
 129   │         value: "true"
 130   │       volumes:
 131   │       - configMap:
 132   │           defaultMode: 420
 133   │           name: grafana-operated-ini
 134   │         name: grafana-operated-ini
 135   │       - emptyDir: {}
 136   │         name: grafana-logs
 137   │       - name: grafana-data
 138   │         persistentVolumeClaim:
 139   │           claimName: grafana-operated-pvc
 140   │ status:
 141   │   observedGeneration: 299
 142   │   replicas: 0
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

theSuess commented 1 month ago

Okay so the issue is definitely in the way the plugins are applied. You can see this by comparing the GF_INSTALL_PLUGINS values across the different replica sets. They should always be sorted the same way

grafana / grafana-operator

[Bug] Operator randomly tries to scale up and down Grafana operated service #1651