search

weaveworks / grafanalib

Python library for building Grafana dashboards
Apache License 2.0
1.86k stars 309 forks source link

Auto generate from grafana schema? #607

Open fzyzcjy opened 1 year ago

fzyzcjy commented 1 year ago

Hi thanks for the lib! I wonder whether it is reasonable to be auto-generated from the grafana schema https://github.com/grafana/grafana/blob/main/kinds/dashboard/dashboard_kind.cue. By doing that, we can, for example:

  1. Always be up-to-date, no missing fields
  2. https://github.com/weaveworks/grafanalib/issues/608 will be much easier
alkuzad commented 1 year ago

Grafana CUE/Thema is cute but highly experimental and not working. What you linked does not even have panel schemas, just general dashboard one Correction, it has them. The whole reason (I believe) grafanalib exists is that jsonnet is super hard to write and almost noone did. Replacements are not yet ready even if hints were given like 1,5year ago that this is coming.

fzyzcjy commented 1 year ago

@alkuzad Thank you for the information. I have implemented (since hours ago) using auto-generation and minor manual modifications: https://github.com/fzyzcjy/grafana-dashboard-python

The whole reason (I believe) grafanalib exists is that jsonnet is super hard to write and almost noone did. Replacements are not yet ready even if hints were given like 1,5year ago that this is coming.

I also do not use jsonnet, and love to use existing toolkits around python (huge ecosystem!)

alkuzad commented 1 year ago

@fzyzcjy It is a nice experiment but I do not believe in the stage for use yet, given the amount of manual changes needed. I saw also one additional problem - I use InfluxDB but Grafana/Grok does not have it and Grafanalib does.

fzyzcjy commented 1 year ago

I do not believe in the stage for use yet, given the amount of manual changes needed

Yes I think so.

I use InfluxDB but Grafana/Grok does not have it and Grafanalib does.

Hmm, where is that schema from? Is it by manual reverse engineering, or the JSON schema is published outside grok?

alkuzad commented 1 year ago

Hmm, where is that schema from? Is it by manual reverse engineering, or the JSON schema is published outside grok?

I just think it was not ported from the older grafonnet-lib. https://github.com/grafana/grafonnet-lib/blob/master/grafonnet/influxdb.libsonnet

fzyzcjy commented 1 year ago

I see. But I guess it is very easy to solve, if all the code is in https://github.com/weaveworks/grafanalib/blob/main/grafanalib/influxdb.py.

We only need to write a Pydantic class Influxdb mimicking that - which can be done in 5 minutes :)

fzyzcjy commented 1 year ago

but I do not believe in the stage for use yet, given the amount of manual changes needed

Btw what manual changes do you have in mind? I have already changed many things, but most are just adding default values and making type constraints looser, so they are quite straightforward.

alkuzad commented 1 year ago

but I do not believe in the stage for use yet, given the amount of manual changes needed

Btw what manual changes do you have in mind? I have already changed many things, but most are just adding default values and making type constraints looser, so they are quite straightforward.

https://github.com/fzyzcjy/grafana-dashboard-python/blob/master/misc/model_generation/manual_patch.patch this, it looked to me like a lot of tuning applied over auto-generated schema.

fzyzcjy commented 1 year ago

Yes and no. It is mostly just

  1. adding default values, s.t. no need to specify a ton of fields when using it. (but it is fairly easy to PR and fix if you see a field that is mandatory but is reasonable to have a default value)
  2. loosing types, s.t. it allows more types than defined in schema. (also trivial to PR, since if you see a type gets rejected but you think it should be good, just change it from SomeType to Union[SomeType, SomeMoreType] and PR)
fzyzcjy commented 1 year ago

@alkuzad By the way, did you find Grafanalib missing some fields once in a while? I did have that problem sometimes, and has to modify its source code.

alkuzad commented 1 year ago

@fzyzcjy I am not an extensive user yet, I am experimenting with dashboard.json generation and especially converting it from the raw export. trimDefaults seems to not work and I want to figure out how to convert dashboard.json raw form into more easy to read one :)

K-Phoen commented 3 months ago

As a quick side-note: a library fully-generated from schemas is being worked on by Grafana and published here https://github.com/grafana/grafana-foundation-sdk/ :)