POC: Generate jsonnet library from upstream schema

[Duologic]

What's this?

Upstream on github.com/grafana/grafana, work is in progress to provide a schema (in CUE) for Grafana objects. This PR takes a first pass at providing a jsonnet library based on this schema.

How does it work?

The script bin/mkschema.sh clones a specific Grafana version and uses Thema to generate a Json Schema (OpenAPI schema actually). It then uses CRDsonnet to convert this schema into a library with jsonnet functions. In main.libsonnet, we also add a few extra shortcuts in addition to the generated functions. In example.libsonnet we use this library to generate in simple dashboard.

What does it give me?

It is still early days for the schema, the upstream schema currently limited to the dashboard schema. And even in that there are lots of missing bits and pieces that are currently covered for much better in the existing Grafonnet, this makes this POC a lot less useful. To make this more useful, we need at least schemas for the different built-in panels and datasource targets.

This is actually a second pass for me personally, I played with this almost a year ago and am currently a bit dissapointed that there was almost no progress in schematizing other compontents (panels/datasources/plugins) since then. I might be wrong on this end, so please tell me if I'm missing something here.

Why is this important?

We get very regular questions from Grafana users and customers on different channels what the best way is to compose dashboards as-code. Grafonnet is currently our best offer we can give them, but it is horribly out-of-date and undermaintained. With the upstream schemas at the horizon, this is unlikely going to improve.

With an upstream schema that gets maintained along with Grafana itself, Grafonnet can offer an up-to-date library that can be maintained with very little effort.

[sdboyer]

Amazing to see this!

We've got about ten panel plugins schematized, though no datasource plugins yet. However, the pipeline is built, so as soon as the respective teams get their schemas written, they'll come rolling in automatically.

It's a lot of chicken-or-egg, but being able to show the outputs - such as this, here - is crucial to demonstrating the value of actually getting the schemas written. Which, in turn, makes it easier to prioritize the schematization work division-wide.

The primary interface core grafana is presenting to all the schemas is for importing them from Go. i've taken some care to make it possible to do what you've done here with raw thema tooling, but it's not the main intended use pattern, so it's gonna be more awkward.

However, now that i can see the kind of output pattern you're going for, i can throw together what should be a fairly trivial Go program (LoC-wise) that generates equivalent output!

[sdboyer]

OK, i spent a while on this yesterday, and discovered quite the frustrating bug in the course of getting the Go code pushed up. i've made some inline comments for explanation on the Go code itself.

Overall, my take is that, while it's possible and (IMO) roughly equivalent ergonomics using thema from bash scripts directly here for now, longer-term the Go approach is likely to be valuable, because there's going to be information from the kind system that just isn't accessible to generic tools.

[sdboyer]

General agreement @Duologic and i reached in backchannel is that the Go logic to generate JSON Schema here is probably something we can do in grok (and be owned by @grafana/grafana-as-code), and then this can just depend on grok for those generated files.

That fits with the larger pattern grok is aiming to fill, anyway!

[Duologic]

POC has served its purpose, closing this.