Adding custom configmap to Grafana deployment

lanmarti commented 4 years ago

What did you do? I'm trying to add a configmap to the grafana deployment, without modifying the original /vendor/grafana/grafana.libsonnet file. However, due to inexperience with jsonnet/ksonnet, I seem to be stuck. By trying to add a new volume, I'm actually overwriting the existing volumes attached to the deployment. I'm looking for a way to add to rather than overwrite the existing volumes, as well as add a mount to the volume mounts of the grafana container that is already defined in grafana.libsonnet.

Did you expect to see some different? Ideally, I would be able to mount the custom config map in the grafana container without modifying grafana.libsonnet itself.

Manifests: I added this block to my own custom.jsonnet file

grafana+:: {
  local deployment = k.apps.v1.deployment;
  local volume = deployment.mixin.spec.template.spec.volumesType;
  local customVolume = volume.withName(volumeName) + volume.mixin.configMap.withName(configMapName);
  deployment.mixin.spec.template.spec.withVolumes(customVolume),
  // TODO add volume mount to container
},

Relevant parts from grafana.libsonnet

// [...]
grafana+: {
// [...]
deployment:
  local deployment = k.apps.v1.deployment;
  local container = k.apps.v1.deployment.mixin.spec.template.spec.containersType;
  local volume = k.apps.v1.deployment.mixin.spec.template.spec.volumesType;
  // [...]

  local configVolumeName = 'grafana-config';
  local configSecretName = 'grafana-config';
  local configVolume = volume.withName(configVolumeName) + volume.mixin.secret.withSecretName(configSecretName);
  local configVolumeMount = containerVolumeMount.new(configVolumeName, '/etc/grafana');

  local storageVolumeName = 'grafana-storage';
  local storageVolume = volume.fromEmptyDir(storageVolumeName);
  local storageVolumeMount = containerVolumeMount.new(storageVolumeName, '/var/lib/grafana');

  local datasourcesVolumeName = 'grafana-datasources';
  local datasourcesSecretName = 'grafana-datasources';
  local datasourcesVolume = volume.withName(datasourcesVolumeName) + volume.mixin.secret.withSecretName(datasourcesSecretName);         local datasourcesVolumeMount = containerVolumeMount.new(datasourcesVolumeName, '/etc/grafana/provisioning/datasources');        
  local dashboardsVolumeName = 'grafana-dashboards';
  local dashboardsConfigMapName = 'grafana-dashboards';
  local dashboardsVolume = volume.withName(dashboardsVolumeName) + volume.mixin.configMap.withName(dashboardsConfigMapName);            local dashboardsVolumeMount = containerVolumeMount.new(dashboardsVolumeName, '/etc/grafana/provisioning/dashboards');

  local volumeMounts =
    [
      storageVolumeMount,
      datasourcesVolumeMount,
      dashboardsVolumeMount,
    ] +
    [
      local dashboardName = std.strReplace(name, '.json', '');
      containerVolumeMount.new('grafana-dashboard-' + dashboardName, '/grafana-dashboard-definitions/0/' + dashboardName)
      for name in std.objectFields($._config.grafana.dashboards)
    ] +
    [
      local dashboardName = std.strReplace(name, '.json', '');
      containerVolumeMount.new('grafana-dashboard-' + dashboardName, '/grafana-dashboard-definitions/0/' + dashboardName)
      for name in std.objectFields($._config.grafana.rawDashboards)
    ] +

    if std.length($._config.grafana.config) > 0 then [configVolumeMount] else [];

  local volumes =
    [
      storageVolume,
      datasourcesVolume,
      dashboardsVolume,
    ] +
    [
      local dashboardName = 'grafana-dashboard-' + std.strReplace(name, '.json', '');
      volume.withName(dashboardName) +
      volume.mixin.configMap.withName(dashboardName)
      for name in std.objectFields($._config.grafana.dashboards)
    ] +
    [
      local dashboardName = 'grafana-dashboard-' + std.strReplace(name, '.json', '');
      volume.withName(dashboardName) +
      volume.mixin.configMap.withName(dashboardName)
      for name in std.objectFields($._config.grafana.rawDashboards)
    ] +
    if std.length($._config.grafana.config) > 0 then [configVolume] else [];

  local c =
    container.new('grafana', $._config.imageRepos.grafana + ':' + $._config.versions.grafana) +
    (if std.length($._config.grafana.plugins) == 0 then {} else container.withEnv([env.new('GF_INSTALL_PLUGINS', std.join(',', $._config.grafana.plugins))])) +
    container.withVolumeMounts(volumeMounts) +
    container.withPorts(containerPort.newNamed(targetPort, portName)) +
    container.mixin.readinessProbe.httpGet.withPath('/api/health') +
    container.mixin.readinessProbe.httpGet.withPort(portName) +
    container.mixin.resources.withRequests($._config.grafana.container.requests) +
    container.mixin.resources.withLimits($._config.grafana.container.limits);

  deployment.new('grafana', 1, c, podLabels) +
  deployment.mixin.metadata.withNamespace($._config.namespace) +
  deployment.mixin.metadata.withLabels(podLabels) +
  deployment.mixin.spec.selector.withMatchLabels(podLabels) +
  deployment.mixin.spec.template.spec.withNodeSelector({ 'beta.kubernetes.io/os': 'linux' }) +
  deployment.mixin.spec.template.spec.withVolumes(volumes) +
  deployment.mixin.spec.template.spec.securityContext.withRunAsNonRoot(true) +
  deployment.mixin.spec.template.spec.securityContext.withRunAsUser(65534) +
  deployment.mixin.spec.template.spec.withServiceAccountName('grafana'),
},

Anything else we need to know?: I'm not trying to add a new dashboard, the configmap contains a file needed for a cluster-specific problem.

brancz commented 4 years ago

I'm not trying to add a new dashboard, the configmap contains a file needed for a cluster-specific problem.

Could you elaborate what it is that you are trying to solve?

metalmatze commented 4 years ago

Usually, add a new dashboard works by adding a new object to the global grafanaDashboards object. As we loop over that object to generate the configmaps and volumes those object are added as dashboards automatically.

Basically:

{
  grafanaDashboards+:: {
    'mydashboard.json':
    ...
  }
}

lanmarti commented 4 years ago

I'm adding an nsswitch.conf file containing hosts: files dns so that host aliases are resolved in the correct order. I'm not fully clear about the details, but basically, some go/alpine based containers ignore the hosts file as set by Kubernetes when using hostAlias specifications in the deployment .yaml, unless the nsswitch.conf file is added. See for example https://github.com/golang/go/issues/22846

I have to use host aliases to make sure the authentication setup for this cluster works correctly. Manually updating the grafana deployment yaml works for now, but obviously adding a few lines to my .jsonnet file is vastly preferred.

lanmarti commented 4 years ago

@metalmatze thanks for your reply, but this is specifically NOT about dashboards, but a separate configMap, see my previous reply. I already have some custom dashboards up and running, indeed through grafanaDashboards, which works like a charm!

brancz commented 4 years ago

I think this is something that is pretty specific to your environment so I would suggest you to do a patch specific to your problem. A jsonnet patch for this would look something like this:

{
   grafana+:: {
    deployment+:
      {
        spec+: {
          template+: {
            spec+: {
              volumes+: [ * volume definition * ],
              containers:
                std.map(
                  function(c)
                    if c.name == 'grafana' then
                      c {
                        volumeMounts+: [
                          * volume mount definition *,
                        ],
                      }
                    else
                      c,
                  super.containers,
                ),
            },
          },
        },
      },
  },
}

lanmarti commented 4 years ago

Thank you, that did the trick!

prometheus-operator / kube-prometheus

Adding custom configmap to Grafana deployment #364