[grafana] DataSource and Alerts provisioning with HTTPS enabled not possible.

I'm using latest Helm chart for Grafana installation on Kubernetes.

I'm using sidecars for alerts, dashboards and datasource provisioning. Data source comes from the secret and appears as a yaml file in Grafana container under /etc/grafana/provisioning/datasources File itself:

apiVersion: 1
datasources:
  - name: Thanos
    access: proxy
    uid: thanos
    orgId: 1
    editable: true
    readOnly: false
    isDefault: true
    jsonData:
grafana-6cb548fb57-9qxvq:/etc/grafana/provisioning/datasources$ cat datasources.yaml
apiVersion: 1
datasources:
  - name: Thanos
    access: proxy
    uid: thanos
    orgId: 1
    editable: true
    readOnly: false
    isDefault: true
    jsonData:
      httpHeaderName1: Authorization
      timeInterval: 5s
      tlsSkipVerify: true
    secureJsonData:
      httpHeaderValue1: 'Bearer XXXX'
    type: prometheus
    url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
    typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg

grafana.ini:

  grafana.ini: |-
    [analytics]
    enabled: false
    check_for_updates=false
    reporting_enabled=false
    feedback_links_enabled=false
    [server]
    http_addr=''
    http_port='3000'
    cert_key='/var/lib/grafana/ssl/tls.key'
    cert_file='/var/lib/grafana/ssl/tls.crt'
    enforce_domain=False
    protocol=https
    protocol="https"
    enable_gzip=true
    [grafana_net]
    url=https://grafana.net
    [log]
    mode=console
    level=info
    [log.console]
    level=info
    [security]
    cookie_secure=true
    [plugins]
    public_key_retrieval_disabled=true
    [auth]
    disable_login_form=false
    disable_signout_menu=True
    [auth.anonymous]
    enabled=false
    [auth.basic]
    enabled=true # Enabled since we need this for API access to avoid manual creation of Grafana Service Accounts
    [auth.generic_oauth]
    allow_sign_up=true
    enabled=true
    name="keycloak"
    client_id="grafana"
    scopes="openid profile email"
    tls_client_ca="/var/lib/grafana/ca/ca-bundle.pem"
    auth_url="https://{{ .Values.auth }}/realms/dp3/protocol/openid-connect/auth"
    token_url="https://{{ .Values.auth }}/realms/dp3/protocol/openid-connect/token"
    api_url="https://{{ .Values.auth }}/realms/dp3/protocol/openid-connect/userinfo"
    role_attribute_path="contains(groups[*], 'platformadmin') && 'GrafanaAdmin' || 'Viewer'"
    [database]
    type="postgres"
    host="postgres-primary.services.svc:5432"
    name="grafana"
    user="grafana"
    ssl_mode="require"
    ca_cert_path="/var/lib/grafana/ssl/ca.crt"

I see that within container start Grafana provisioning alerts:

$ oc logs -f grafana-6cb548fb57-9qxvq | grep -i provisioning
logger=settings t=2024-02-12T12:59:42.133795502Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2024-02-12T12:59:42.13416154Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2024-02-12T12:59:42.134955794Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=provisioning.alerting t=2024-02-12T12:59:42.573683314Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2024-02-12T12:59:42.573826465Z level=info msg="finished to provision alerting"

But I don't see any errors with Data Source provisioning. If I shell into container and start grafana server there it detects datasources.yaml and provisions it without any single error or warning.

What do I miss here? How can I provision Data Sources from secrets with Helm chart?

I see that dashboard provisioning works differently and they are stored as jsons under /tmp/dashboards/*.json. May this be related?

grafana / helm-charts

[grafana] DataSource and Alerts provisioning with HTTPS enabled not possible. #2962