Provisioning via YAML

[chris-bosman]

Our Grafana instance is running in a Kubernetes cluster and, as such, I'm attempting to create ConfigMap resources that will import all necessary dashboards & datasources at creation. I'm running into an issue trying to import this particular datasource.

In querying the API, I was able to glean the following data when I manually added the Azure Monitor datasources:

[
    {
        "id":2,
        "orgId":1,
        "name":"<Name>",
        "type":"grafana-azure-monitor-datasource",
        "typeLogoUrl":"public/plugins/grafana-azure-monitor-datasource/img/logo.jpg",
        "access":"proxy",
        "url":"",
        "password":"",
        "user":"",
        "database":"",
        "basicAuth":false,
        "isDefault":false,
        "jsonData":
            {
                "appInsightsAppId":"",
                "clientId":"<clientId>",
                "cloudName":"azuremonitor",
                "keepCookies":[],
                "subscriptionId":"<subscriptionId>",
                "tenantId":"<tenantId>"},
                "readOnly":false
            },
    {
        "id":6,
        "orgId":1,
        "name":"<name>",
        "type":"grafana-azure-monitor-datasource",
        "typeLogoUrl":"public/plugins/grafana-azure-monitor-datasource/img/logo.jpg",
        "access":"proxy",
        "url":"",
        "password":"",
        "user":"",
        "database":"",
        "basicAuth":false,
        "isDefault":false,
        "jsonData":
            {
                "appInsightsAppId":"<appInsightsAppId>",
                "cloudName":"azuremonitor",
                "keepCookies":[]
            },
        "readOnly":false
    }
]

I used this as the guide for my datasource YAML file. Additionally, I made the assumption that the other required values (namely, clientSecret for the Azure Monitor datasource and appInsightsApiKey for the App Insights datasource) would be imported via secureJsonData. Based off that, I came up with the following data for a ConfigMap (within a Helm chart):

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Values.app.name }}-datasource
  labels:
    grafana_datasources: "1"
data:
  {{ .Values.app.name }}-datasource.yaml: |-
    datasources:
    - name: <name>
      type: grafana-azure-monitor-datasource
      access: proxy
      orgId: 1
      url:
      isDefault: true
      jsonData:
        cloudName: azuremonitor
        clientId: {{ .Values.azure.clientId | b64dec }}
        subscriptionId: {{ .Values.azure.subscriptionId | b64dec }}
        tenantId: {{ .Values.azure.tenantId | b64dec }}
      secureJsonData:
        clientSecret: {{ .Values.azure.clientSecret | b64dec }}
      editable: true
    - name: <name>
      type: grafana-azure-monitor-datasource
      access: proxy
      orgId: 1
      url: 
      isDefault: false
      jsonData:
        cloudName: azuremonitor
        appInsightsAppId: {{ .Values.azure.appInsightsAppId | b64dec }}
      secureJsonData:
        appInsightsApiKey: {{ .Values.azure.appInsightsApiKey | b64dec }}
      editable: true

This ConfigMap gets properly created and input into my Kubernetes cluster. Additionally, the values I see in the YAML file are the approriate and expected values. However, in Grafana itself, while my datasources exist, their values are not populated. This includes both the jsonData and the secureJsonData.

Can someone offer any insight into why my values are not flowing from my YAML to my datasource? Thanks!

[chris-bosman]

Any thoughts on this?

[dougnd]

Hmm. I also wanted to be able to provision this datasource and tried to reverse engineer the required fields. The following worked for me:

apiVersion: 1

datasources:
  - name: Azure
    type: grafana-azure-monitor-datasource
    orgId: 1
    typeLogoUrl: public/plugins/grafana-azure-monitor-datasource/img/logo.jpg
    access: proxy
    url: /api/datasources/proxy/2
    isDefault: true
    jsonData:
      cloudName: azuremonitor
      subscriptionId: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
      tenantId: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
      clientId: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
      logAnalyticsDefaultWorkspace: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
      azureLogAnalyticsSameAs: true
      keepCookies: []
    secureJsonData:
      clientSecret: LOG_ANALYTICS_CLIENT_SECRET
    editable: true

(I am using the log-analytics branch)

I don't see any major difference so I do not know why it would not work.

[cschyma]

I guess I am trying to achieve the same. Importing Data Sources at Grafana start. Unfortunately I haven't found any documentation on providing Azure Monitor Client Secret and Application Insights API Key as well.

My JSON to import looks like the following:

{
    "id": 3,
    "orgId": 1,
    "name": "Datasource X",
    "type": "grafana-azure-monitor-datasource",
    "typeLogoUrl": "",
    "access": "proxy",
    "url": "/api/datasources/proxy/3",
    "password": "",
    "user": "",
    "database": "",
    "basicAuth": false,
    "basicAuthUser": "",
    "basicAuthPassword": "",
    "withCredentials": false,
    "isDefault": false,
    "jsonData": {
        "appInsightsAppId": "id",
        "azureLogAnalyticsSameAs": false,
        "clientId": "clientid",
        "cloudName": "azuremonitor",
        "keepCookies": [],
        "subscriptionId": "subid",
        "tenantId": "tenantid"
    },
    "secureJsonFields": {
        "appInsightsApiKey": true,
        "clientSecret": true
    },
    "version": 5,
    "readOnly": false
}

Overwriting appInsightsApiKey and clientSecret does not work, The web ui field keep being empty.

[dougnd]

What versions of Grafana and Azure Monitor Datasource are you using? Things seem to work fine for me with the latest versions in docker. For example, with this Dockerfile:

FROM grafana/grafana:latest

COPY ./datasources.yaml /etc/grafana/provisioning/datasources

USER root
RUN chown -R grafana:grafana /etc/grafana/provisioning
USER grafana

and this datasources.yaml

apiVersion: 1

datasources:
  - name: Azure
    type: grafana-azure-monitor-datasource
    orgId: 1
    typeLogoUrl: public/plugins/grafana-azure-monitor-datasource/img/logo.jpg
    access: proxy
    url: /api/datasources/proxy/2
    isDefault: true
    jsonData:
      cloudName: azuremonitor
      subscriptionId: 11111111-1111-1111-1111-111111111111
      tenantId: 11111111-1111-1111-1111-111111111111
      clientId: 11111111-1111-1111-1111-111111111111
      logAnalyticsDefaultWorkspace: 11111111-1111-1111-1111-111111111111
      azureLogAnalyticsSameAs: true
      keepCookies: []
    secureJsonData:
      clientSecret: 11111111-1111-1111-1111-111111111111
    editable: true

in the same directory, I can run

docker build -t azure-monitor-test .
docker run -d -p 3000:3000 -e "GF_INSTALL_PLUGINS=grafana-azure-monitor-datasource" azure-monitor-test

Then open the Grafana instance at localhost:3000 in a browser. I can see the a Azure datasource with information filled in:

Hopefully this helps provide some known working reference point.

[chris-bosman]

@dougnd The only difference I see in our configs is that you've explicitly set the URL, whereas I left my URL field blank. I'll be trying to explicitly set this on my next attempt and will report back.

[jungopro]

Hi @chris-bosman & @dougnd, I stumbled upon this when I was looking for a way to do the provisioning. I was able to make it work using the following code in Kubernetes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: azure-configmap
data:  
  azure-datasource.json: |+
    {
      "name": "Test-Azure",
      "access": "proxy",
      "basicAuth": false,
      "type": "grafana-azure-monitor-datasource",
      "url": "/api/datasources/proxy/2",
      "isDefault": false,
      "jsonData": {
        "cloudName": "azuremonitor",
        "subscriptionId": "...",
        "tenantId": "...",
        "clientId": "...",
        "azureLogAnalyticsSameAs": true,
        "logAnalyticsDefaultWorkspace": "..."
      },
      "secureJsonData": {
        "clientSecret": "..."
      }
    }

btw, I'm using kube-prometheus to deploy both prometheus and grafana. The grafana version in that chart is 5.0.0. I don't think it will make any difference as long as you keep the same concept that:

• the config map will create a yaml in a folder that grafana is watching
• the yaml file will contain json data (as in the example)

I hope this helps, let me know if you need any further assistance, I will be happy to assist