search

grafana / grafana-operator

An operator for Grafana that installs and manages Grafana instances, Dashboards and Datasources through Kubernetes/OpenShift CRs
https://grafana.github.io/grafana-operator/
Apache License 2.0
916 stars 397 forks source link

[Bug] GrafanaDashboard is not getting created #740

Closed lucamaf closed 2 years ago

lucamaf commented 2 years ago

Describe the bug Even if I created the object GrafanaDashboard this is not being created inside Grafana

Version Operator being used 4.3.0

To Reproduce Steps to reproduce the behavior:

  1. Install version 4.3.0 of Grafana Operator
  2. Create a Grafana Object
  3. Create a GrafanaDashboard using JSON spec
  4. See Grafana Dashboard not being created inside Grafana

Expected behavior Expected to see Grafana dashboard inside Grafana instance

Screenshots If applicable, add screenshots to help explain your problem.

Runtime (please complete the following information):

Additional context Add any other context about the problem here.

nemerna commented 2 years ago

I am facing Similar Issue with my grafana-operator V4.3.0 using the early version 3.x i can create dashboards and see them in grafana

NissesSenap commented 2 years ago

@lucamaf and @nemerna how does your grafana yaml look like? I have written documentation trying to do some basic debugging around this. Please read: https://github.com/grafana-operator/grafana-operator/blob/master/documentation/debug.md

In general my greatest tip is to do everything as basic as possible and then add more fancy grafana features after you have seen that the dashboards are getting imported as intended.

Grafana example: https://github.com/grafana-operator/grafana-operator/blob/master/deploy/examples/Grafana.yaml

Dashboard example: https://github.com/grafana-operator/grafana-operator/tree/master/deploy/examples/dashboards

If you solve the issue please share what steps you took to make it easier for other people to learn.

pb82 commented 2 years ago

@lucamaf can you share your dashboard CR?

giusepperaso commented 2 years ago

I have the same problem, I am trying with the exact same examples in the repo (only the namespace is different) but the dashboard is not visible:

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: example-grafana
  namespace: monitoring
spec:
  ingress:
    enabled: True
    hostname: "grafana.apps.127.0.0.1.nip.io"
    path: /
    pathType: Prefix
  config:
    log:
      mode: "console"
      level: "warn"
    security:
      admin_user: "root"
      admin_password: "secret"
    auth:
      disable_login_form: False
      disable_signout_menu: True
    auth.anonymous:
      enabled: True
  dashboardLabelSelector:
    - matchExpressions:
        - { key: app, operator: In, values: [grafana] }
---
apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: simple-dashboard
  namespace: monitoring
  labels:
    app: grafana
spec:
  json: >
    {
      "id": null,
      "title": "Simple Dashboard",
      "tags": [],
      "style": "dark",
      "timezone": "browser",
      "editable": true,
      "hideControls": false,
      "graphTooltip": 1,
      "panels": [],
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "timepicker": {
        "time_options": [],
        "refresh_intervals": []
      },
      "templating": {
        "list": []
      },
      "annotations": {
        "list": []
      },
      "refresh": "5s",
      "schemaVersion": 17,
      "version": 0,
      "links": []
    }

When I go in the logs, I see this problem:

INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}

But also:

controller-runtime.manager      no grafana instance available

Complete logs:

Found 2 pods, using pod/grafana-grafana-operator-9b8c99d9-frjmc
I0522 04:36:09.976408       1 request.go:655] Throttling request took 1.035299s, request: GET:https://10.43.0.1:443/apis/monitoring.coreos.com/v1?timeout=32s
2022-05-22T04:36:10.235Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2022-05-22T04:36:10.237Z        INFO    leader  Trying to become the leader.
2022-05-22T04:36:11.540Z        DEBUG   leader  Found podname   {"Pod.Name": "grafana-grafana-operator-9b8c99d9-frjmc"}
2022-05-22T04:36:11.543Z        DEBUG   leader  Found Pod       {"Pod.Namespace": "monitoring", "Pod.Name": "grafana-grafana-operator-9b8c99d9-frjmc"}
2022-05-22T04:36:11.546Z        INFO    leader  No pre-existing lock was found.
2022-05-22T04:36:11.549Z        INFO    leader  Became the leader.
2022-05-22T04:36:11.549Z        INFO    Registering Components.
2022-05-22T04:36:11.550Z        INFO    Starting the Cmd.
2022-05-22T04:36:12.853Z        INFO    Starting dashboard controller
2022-05-22T04:36:12.853Z        INFO    controller-runtime.manager.controller.grafanadashboard  Starting EventSource    {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDashboard", "source": "kind source: /, Kind="}
2022-05-22T04:36:12.853Z        INFO    controller-runtime.manager.controller.grafanadashboard-controller       Starting EventSource    {"source": "kind source: /, Kind="}
2022-05-22T04:36:12.953Z        INFO    controller-runtime.manager.controller.grafanadashboard-controller       Starting Controller
2022-05-22T04:36:12.953Z        INFO    controller-runtime.manager.controller.grafanadashboard  Starting Controller     {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDashboard"}
2022-05-22T04:36:12.953Z        INFO    controller-runtime.manager.controller.grafanadashboard-controller       Starting workers        {"worker count": 1}
2022-05-22T04:36:12.953Z        INFO    controller-runtime.manager.controller.grafanadashboard  Starting workers        {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDashboard", "worker count": 1}
2022-05-22T04:36:14.156Z        INFO    Starting notificationchannel controller
2022-05-22T04:36:14.156Z        INFO    setup   starting manager with options   {"watchNamespace": "monitoring", "dashboardNamespaces": "", "scanAll": false}
2022-05-22T04:36:14.156Z        INFO    controller-runtime.manager.controller.grafananotificationchannel-controller     Starting EventSource    {"source": "kind source: /, Kind="}
I0522 04:36:14.156595       1 leaderelection.go:243] attempting to acquire leader lease monitoring/2c0156f0.integreatly.org...
2022-05-22T04:36:14.156Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
I0522 04:36:14.164994       1 leaderelection.go:253] successfully acquired lease monitoring/2c0156f0.integreatly.org
2022-05-22T04:36:14.165Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"ConfigMap","namespace":"monitoring","name":"2c0156f0.integreatly.org","uid":"f6f583ac-a992-4316-9369-31c20cf3f080","apiVersion":"v1","resourceVersion":"1839"}, "reason": "LeaderElection", "message": "grafana-grafana-operator-9b8c99d9-frjmc_efb6f1f4-98fb-4b2d-972a-9757ab205016 became leader"}
2022-05-22T04:36:14.165Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"Lease","namespace":"monitoring","name":"2c0156f0.integreatly.org","uid":"a3f7009e-422d-4cf2-925e-5fd8ceedef77","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1840"}, "reason": "LeaderElection", "message": "grafana-grafana-operator-9b8c99d9-frjmc_efb6f1f4-98fb-4b2d-972a-9757ab205016 became leader"}
2022-05-22T04:36:14.165Z        INFO    controller-runtime.manager.controller.grafanadatasource Starting EventSource    {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDataSource", "source": "kind source: /, Kind="}
2022-05-22T04:36:14.165Z        INFO    controller-runtime.manager.controller.grafanadashboard  Starting EventSource    {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDashboard", "source": "kind source: /, Kind="}
2022-05-22T04:36:14.165Z        INFO    controller-runtime.manager.controller.grafana   Starting EventSource    {"reconciler group": "integreatly.org", "reconciler kind": "Grafana", "source": "kind source: /, Kind="}
2022-05-22T04:36:14.257Z        INFO    controller-runtime.manager.controller.grafananotificationchannel-controller     Starting Controller
2022-05-22T04:36:14.258Z        INFO    controller-runtime.manager.controller.grafananotificationchannel-controller     Starting workers        {"worker count": 1}
2022-05-22T04:36:14.265Z        INFO    controller-runtime.manager.controller.grafana   Starting EventSource    {"reconciler group": "integreatly.org", "reconciler kind": "Grafana", "source": "kind source: /, Kind="}
2022-05-22T04:36:14.265Z        INFO    controller-runtime.manager.controller.grafanadashboard  Starting Controller     {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDashboard"}
2022-05-22T04:36:14.265Z        INFO    controller-runtime.manager.controller.grafanadatasource Starting EventSource    {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDataSource", "source": "kind source: /, Kind="}
2022-05-22T04:36:14.265Z        INFO    controller-runtime.manager.controller.grafana   Starting Controller     {"reconciler group": "integreatly.org", "reconciler kind": "Grafana"}
2022-05-22T04:36:14.265Z        INFO    controller-runtime.manager.controller.grafanadashboard  Starting workers        {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDashboard", "worker count": 1}
2022-05-22T04:36:14.366Z        INFO    controller-runtime.manager.controller.grafanadatasource Starting Controller     {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDataSource"}
2022-05-22T04:36:14.366Z        INFO    controller-runtime.manager.controller.grafanadatasource Starting workers        {"reconciler group": "integreatly.org", "reconciler kind": "GrafanaDataSource", "worker count": 1}
2022-05-22T04:36:14.366Z        INFO    controller-runtime.manager.controller.grafana   Starting workers        {"reconciler group": "integreatly.org", "reconciler kind": "Grafana", "worker count": 1}
2022-05-22T04:36:14.872Z        DEBUG   action-runner   (    0)    SUCCESS create admin credentials secret
2022-05-22T04:36:14.878Z        DEBUG   action-runner   (    1)    SUCCESS create grafana service
2022-05-22T04:36:14.882Z        DEBUG   action-runner   (    2)    SUCCESS create grafana service account
2022-05-22T04:36:14.885Z        DEBUG   action-runner   (    3)    SUCCESS create grafana config
2022-05-22T04:36:14.888Z        DEBUG   action-runner   (    4)    SUCCESS create grafanadatasource config
2022-05-22T04:36:14.888Z        DEBUG   action-runner   (    5)    SUCCESS plugins unchanged
2022-05-22T04:36:14.893Z        DEBUG   action-runner   (    6)    SUCCESS create grafana deployment
2022-05-22T04:36:14.893Z        DEBUG   action-runner   (    7)     FAILED check deployment readiness
2022-05-22T04:36:14.894Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1139"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:36:14.919Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:36:14.929Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:36:14.932Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:36:14.935Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:36:14.935Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:36:14.947Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:36:14.947Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:36:14.954Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:36:14.959Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:36:14.963Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:36:14.966Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:36:14.969Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:36:14.969Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:36:15.072Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:36:15.072Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:36:15.072Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1875"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:36:15.082Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:36:15.086Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:36:15.088Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:36:15.090Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:36:15.090Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:36:15.096Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:36:15.096Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:36:15.097Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1899"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:36:22.854Z        INFO    running periodic dashboard resync
2022-05-22T04:36:22.854Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:36:24.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:36:24.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:36:24.918Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:36:24.921Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:36:24.923Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:36:24.926Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:36:24.926Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:36:24.931Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:36:24.931Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:36:24.931Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1899"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:36:32.853Z        INFO    running periodic dashboard resync
2022-05-22T04:36:32.853Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:36:34.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:36:34.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:36:34.939Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:36:34.945Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:36:34.951Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:36:34.956Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:36:34.956Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:36:34.972Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:36:34.972Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:36:34.972Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1899"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:36:42.854Z        INFO    running periodic dashboard resync
2022-05-22T04:36:42.854Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:36:44.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:36:44.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:36:44.979Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:36:44.991Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:36:44.995Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:36:45.000Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:36:45.000Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:36:45.016Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:36:45.016Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:36:45.017Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1899"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:36:52.854Z        INFO    running periodic dashboard resync
2022-05-22T04:36:52.854Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:36:54.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:36:54.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:36:55.029Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:36:55.044Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:36:55.048Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:36:55.062Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:36:55.062Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:36:55.076Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:36:55.076Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:36:55.076Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1899"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:37:02.853Z        INFO    running periodic dashboard resync
2022-05-22T04:37:02.853Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:37:04.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:37:04.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:37:05.081Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:05.086Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:05.088Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:05.091Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:05.091Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:05.097Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:05.097Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:37:05.097Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1899"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:37:12.853Z        INFO    running periodic dashboard resync
2022-05-22T04:37:12.853Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:37:14.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:37:14.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:37:15.101Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:15.106Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:15.108Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:15.110Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:15.110Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:15.116Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:15.116Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:37:15.116Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"1899"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:37:22.854Z        INFO    running periodic dashboard resync
2022-05-22T04:37:22.854Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:37:24.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:37:24.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:37:25.120Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:25.123Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:25.125Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:25.128Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:25.128Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:25.133Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:25.133Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:37:25.140Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:37:25.143Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:25.147Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:25.149Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:25.151Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:25.151Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:25.157Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:25.157Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:37:25.157Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:37:31.170Z        DEBUG   controller-runtime.manager      dashboard monitoring/grafana-dashboard-from-url found but selectors do not match        {"controller_grafanadashboard": "monitoring/grafana-dashboard-from-url"}
2022-05-22T04:37:31.170Z        DEBUG   controller-runtime.manager      dashboard monitoring/grafana-dashboard-from-url found but selectors do not match        {"controller_grafanadashboard": "monitoring/grafana-dashboard-from-url"}
2022-05-22T04:37:31.172Z        INFO    controllers.GrafanaDashboard    no grafana instance available   {"controller_grafanadashboard": "monitoring/grafana-dashboard-from-url"}
2022-05-22T04:37:31.196Z        INFO    controllers.GrafanaDashboard    no grafana instance available   {"controller_grafanadashboard": "monitoring/simple-dashboard2"}
2022-05-22T04:37:31.197Z        DEBUG   controller-runtime.manager      dashboard monitoring/simple-dashboard2 found but selectors do not match {"controller_grafanadashboard": "monitoring/simple-dashboard2"}
2022-05-22T04:37:31.197Z        DEBUG   controller-runtime.manager      dashboard monitoring/simple-dashboard2 found but selectors do not match {"controller_grafanadashboard": "monitoring/simple-dashboard2"}
2022-05-22T04:37:31.219Z        INFO    controllers.GrafanaDashboard    no grafana instance available   {"controller_grafanadashboard": "monitoring/simple-dashboard"}
2022-05-22T04:37:31.219Z        DEBUG   controller-runtime.manager      dashboard monitoring/simple-dashboard found but selectors do not match  {"controller_grafanadashboard": "monitoring/simple-dashboard"}
2022-05-22T04:37:31.219Z        DEBUG   controller-runtime.manager      dashboard monitoring/simple-dashboard found but selectors do not match  {"controller_grafanadashboard": "monitoring/simple-dashboard"}
2022-05-22T04:37:31.237Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:37:31.238Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2731"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:37:31.246Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:37:31.246Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:37:31.258Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/istio-mesh-dashboard"}
2022-05-22T04:37:31.258Z        INFO    controllers.GrafanaDashboard    no grafana instance available   {"controller_grafanadashboard": "monitoring/istio-mesh-dashboard"}
2022-05-22T04:37:31.258Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/istio-mesh-dashboard"}
2022-05-22T04:37:31.280Z        INFO    controllers.GrafanaDatasource   datasource successfully imported        {"grafanadatasource": "monitoring/prometheus", "datasource.Namespace": "monitoring", "datasource.Name": "prometheus"}
2022-05-22T04:37:32.853Z        INFO    running periodic dashboard resync
2022-05-22T04:37:32.853Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:37:34.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:37:34.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:37:35.144Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:35.148Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:35.150Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:35.153Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:35.153Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:35.161Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:35.161Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:37:35.161Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:37:41.246Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:37:41.246Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:37:42.853Z        INFO    running periodic dashboard resync
2022-05-22T04:37:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:37:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:37:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:37:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:37:44.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:37:44.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:37:45.166Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:45.173Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:45.177Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:45.180Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:45.180Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:45.191Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:45.191Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:37:45.191Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2521"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:37:45.205Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:45.209Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:45.213Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:45.217Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:45.217Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:45.225Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:45.225Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:37:45.225Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"3030"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:37:51.247Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:37:51.247Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:37:52.854Z        INFO    running periodic dashboard resync
2022-05-22T04:37:52.854Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:37:54.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:37:54.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:37:55.205Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:37:55.210Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:37:55.212Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:37:55.215Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:37:55.215Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:37:55.223Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:37:55.223Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:37:55.223Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"3030"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:38:01.248Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:38:01.248Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:38:02.854Z        INFO    running periodic dashboard resync
2022-05-22T04:38:02.854Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:38:04.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:38:04.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:38:05.228Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:38:05.232Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:38:05.235Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:38:05.239Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:38:05.239Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:38:05.246Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:38:05.246Z        DEBUG   action-runner   (    6)     FAILED check deployment readiness
2022-05-22T04:38:05.246Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-grafana-operator-grafana","uid":"febe3c75-6ac8-4c6a-9ce1-55a636dcb486","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"3030"}, "reason": "ProcessingError", "message": "deployment not ready"}
2022-05-22T04:38:11.249Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:38:11.250Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:38:12.854Z        INFO    running periodic dashboard resync
2022-05-22T04:38:12.854Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafanadashboard": "monitoring/"}
2022-05-22T04:38:14.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:38:14.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:38:15.250Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:38:15.256Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:38:15.258Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:38:15.261Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:38:15.261Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:38:15.266Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:38:15.266Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:38:15.272Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:38:15.276Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:38:15.281Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:38:15.284Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:38:15.286Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:38:15.286Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:38:15.292Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:38:15.293Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:38:15.299Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:38:21.251Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:38:21.251Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:38:22.854Z        INFO    running periodic dashboard resync
2022-05-22T04:38:22.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:38:22.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:38:22.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:38:22.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:38:24.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:38:24.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:38:25.275Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:38:25.279Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:38:25.281Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:38:25.284Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:38:25.284Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:38:25.290Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:38:25.290Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:38:25.296Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:38:31.251Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:38:31.251Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:38:32.854Z        INFO    running periodic dashboard resync
2022-05-22T04:38:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:38:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:38:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:38:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:38:34.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:38:34.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:38:35.299Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:38:35.303Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:38:35.306Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:38:35.308Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:38:35.308Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:38:35.314Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:38:35.314Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:38:35.320Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:38:41.252Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:38:41.252Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:38:42.854Z        INFO    running periodic dashboard resync
2022-05-22T04:38:42.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:38:42.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:38:42.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:38:42.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:38:44.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:38:44.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:38:45.323Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:38:45.328Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:38:45.331Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:38:45.333Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:38:45.333Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:38:45.338Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:38:45.338Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:38:45.345Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:38:51.253Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:38:51.254Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:38:52.853Z        INFO    running periodic dashboard resync
2022-05-22T04:38:52.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:38:52.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:38:52.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:38:52.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:38:54.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:38:54.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:38:55.349Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:38:55.353Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:38:55.356Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:38:55.358Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:38:55.358Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:38:55.364Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:38:55.364Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:38:55.373Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:39:01.255Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:39:01.255Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:39:02.853Z        INFO    running periodic dashboard resync
2022-05-22T04:39:02.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:39:02.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:39:02.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:39:02.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:39:04.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:39:04.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:39:05.377Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:39:05.380Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:39:05.382Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:39:05.385Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:39:05.385Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:39:05.390Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:39:05.390Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:39:05.397Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:39:11.256Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:39:11.256Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:39:12.854Z        INFO    running periodic dashboard resync
2022-05-22T04:39:12.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:39:12.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:39:12.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:39:12.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:39:14.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:39:14.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:39:15.401Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:39:15.407Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:39:15.410Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:39:15.412Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:39:15.412Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:39:15.418Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:39:15.418Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:39:15.424Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:39:21.257Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:39:21.257Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:39:22.853Z        INFO    running periodic dashboard resync
2022-05-22T04:39:22.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:39:22.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:39:22.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:39:22.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:39:24.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:39:24.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:39:25.428Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:39:25.432Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:39:25.434Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:39:25.436Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:39:25.437Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:39:25.442Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:39:25.442Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:39:25.449Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:39:31.258Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:39:31.258Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:39:32.854Z        INFO    running periodic dashboard resync
2022-05-22T04:39:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:39:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:39:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:39:32.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:39:34.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:39:34.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:39:35.452Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:39:35.456Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:39:35.458Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:39:35.460Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:39:35.460Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:39:35.466Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:39:35.466Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:39:35.472Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:39:41.259Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:39:41.259Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:39:42.853Z        INFO    running periodic dashboard resync
2022-05-22T04:39:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:39:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:39:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:39:42.853Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:39:44.156Z        INFO    running periodic notificationchannel resync
2022-05-22T04:39:44.156Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:39:45.478Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:39:45.484Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:39:45.486Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:39:45.489Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:39:45.489Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:39:45.494Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:39:45.494Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:39:45.501Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:39:51.260Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:39:51.260Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:39:52.854Z        INFO    running periodic dashboard resync
2022-05-22T04:39:52.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:39:52.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:39:52.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:39:52.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:39:54.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:39:54.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:39:55.505Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:39:55.508Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:39:55.511Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:39:55.514Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:39:55.514Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:39:55.519Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:39:55.519Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:39:55.526Z        DEBUG   grafana-controller      desired cluster state met
2022-05-22T04:40:01.261Z        DEBUG   action-runner   (    0)     FAILED update admin credentials secret
2022-05-22T04:40:01.261Z        DEBUG   controller-runtime.manager.events       Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"example-grafana","uid":"3107cf43-7765-4e85-8a7f-c1a310ed8698","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"2735"}, "reason": "ProcessingError", "message": "Object monitoring/grafana-admin-credentials is already owned by another Grafana controller grafana-grafana-operator-grafana"}
2022-05-22T04:40:02.854Z        INFO    running periodic dashboard resync
2022-05-22T04:40:02.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard"}
2022-05-22T04:40:02.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "istio-mesh-dashboard"}
2022-05-22T04:40:02.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "grafana-dashboard-from-url"}
2022-05-22T04:40:02.854Z        INFO    dashboard found but selectors do not match      {"namespace": "monitoring", "name": "simple-dashboard2"}
2022-05-22T04:40:04.157Z        INFO    running periodic notificationchannel resync
2022-05-22T04:40:04.157Z        INFO    controller-runtime.manager      no grafana instance available   {"controller_grafananotificationchannel": "monitoring/"}
2022-05-22T04:40:05.529Z        DEBUG   action-runner   (    0)    SUCCESS update admin credentials secret
2022-05-22T04:40:05.533Z        DEBUG   action-runner   (    1)    SUCCESS update grafana service
2022-05-22T04:40:05.535Z        DEBUG   action-runner   (    2)    SUCCESS update grafana service account
2022-05-22T04:40:05.537Z        DEBUG   action-runner   (    3)    SUCCESS update grafana config
2022-05-22T04:40:05.537Z        DEBUG   action-runner   (    4)    SUCCESS plugins unchanged
2022-05-22T04:40:05.543Z        DEBUG   action-runner   (    5)    SUCCESS update grafana deployment
2022-05-22T04:40:05.543Z        DEBUG   action-runner   (    6)    SUCCESS check deployment readiness
2022-05-22T04:40:05.550Z        DEBUG   grafana-controller      desired cluster state met

Version is 4.4.1, in a K3D cluster, installed through bitnami helm 2.5.8

anushanagireddy0430 commented 2 years ago

Similar issue in my case also when running grafana on https enabled. Using grafana v8.0.3, operator v4.1.0. Grafana dashboard CR created successfully, and able to see namespace folder in UI but dashboard is not created inside folder. events found when described dashboard CR. image

Bug Reference: https://github.com/grafana-operator/grafana-operator/issues/642

lucamaf commented 2 years ago

@NissesSenap I tried deploying even the simple one link and not even that one syncs. I'm now on grafana 4.4.1

NissesSenap commented 2 years ago

@lucamaf can you try the same thing as described in this comment: https://github.com/grafana-operator/grafana-operator/issues/680#issuecomment-1129787252

lucamaf commented 2 years ago

that's not possible when you deploy using the community operator on openshift, there is no installation for the operator on the whole cluster

NissesSenap commented 2 years ago

You can always manually try. It would be good to verify that it's the same issue

github-actions[bot] commented 2 years ago

This issue hasn't been updated for a while, marking as stale, please respond within the next 7 days to remove this label