Can't apply prometheus-ksonnet: node-exporter dashboard won't fit into a ConfigMap

[colega]

If I follow the instructions to create prometheus-ksonnet env:

~/tmp 14:50:46
$ mkdir prom-ksonnet-configmap

~/tmp 14:50:53
$ cd prom-ksonnet-configmap 

~/tmp/prom-ksonnet-configmap 14:50:55
$ tk init --k8s=1.24
GET https://github.com/jsonnet-libs/k8s-libsonnet/archive/8ecd8500e820a5ef4e0394f64e383b0edb768137.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/jsonnet-libs/docsonnet/archive/c9cc5534fc5197fb256f323f6b953a7a6bfd9ed6.tar.gz 200
Directory structure set up! Remember to configure the API endpoint:
`tk env set environments/default --server=https://127.0.0.1:6443`

~/tmp/prom-ksonnet-configmap 14:51:25
$ kind get clusters
kind

~/tmp/prom-ksonnet-configmap 14:51:29
$ kcuc kind-kind 
Switched to context "kind-kind".

~/tmp/prom-ksonnet-configmap 14:51:34
$ export CONTEXT=$(kubectl config current-context)

~/tmp/prom-ksonnet-configmap 14:51:39
$ tk env set environments/default  --server-from-context=$CONTEXT
updated spec.apiServer (`` -> `https://127.0.0.1:37469`)
WARNING: version difference between client (1.26) and server (1.21) exceeds the supported minor version skew of +/-1

~/tmp/prom-ksonnet-configmap 14:51:44
$ jb install github.com/grafana/jsonnet-libs/prometheus-ksonnet
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/grafana/grafana/archive/1120f9e255760a3c104b57871fcb91801e934382.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/prometheus/alertmanager/archive/e35e13d7a1cd6eeca983b3688d5faa665f0e498d.tar.gz 200
GET https://github.com/prometheus/prometheus/archive/5583c77b3ae7435fa219a777aa11ce15b8cf83bb.tar.gz 200
GET https://github.com/rfrail3/grafana-dashboards/archive/3cc4366833f36bc7898d6b3505cfda3bd199d58b.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/07ee0700006ecbc8704ff0b2790fccac18c36c6d.tar.gz 200
GET https://github.com/prometheus/node_exporter/archive/c13f8086194f55b1204c3165836a809f2c9aaabe.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/30280196507e0fe6fa978a3e0eaca3a62844f817.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/aaa78e30125378f2aae68fcd29678ea28530e2e1.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/30280196507e0fe6fa978a3e0eaca3a62844f817.tar.gz 200

~/tmp/prom-ksonnet-configmap 14:52:34
$ cat <<EOF > environments/default/main.jsonnet
local prometheus = import "prometheus-ksonnet/prometheus-ksonnet.libsonnet";

prometheus {
  _config+:: {
    cluster_name: "cluster1",
    namespace: "default",
  },
}
EOF

~/tmp/prom-ksonnet-configmap 14:52:36
$ tk apply environments/default
...
[ skipped very long diff ]
...
statefulset.apps/prometheus created
The ConfigMap "dashboards-node-exporter-full-0" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error: exit status 1

This is caused by the node-exporter-full.json dashboard from rfrail13/grafana-dashboards.

A temp fix for this is to disable the node_exporter_full dashboard:

    mixins+:: {
      node_exporter_full: {}, 
    },

[kjamieson-xilis]

Just encountered this as well trying out tanka for prom/grafana.

[nikosmeds]

I recall this being related to Tanka and client-side apply. Can you try applying with

tk apply --apply-strategy=server --force