RESTful API? (Programatic mass-generation of reports)

[senpro-ingwersenk]

Hello there,

Describe the issue I want to automate the process of generating a couple of reports. The way our Grafana is set up is with multi-tenancy in mind where each tenant (= customer) is represented as an org. Now I have figured out how to utilize provisioning with Grafana and can thus inject the same dashboard variables everywhere, but now I would like to automate the process.

Technically, something akin to this: (pseudo-code for simplicity)

orgs: array[Org] = grafana.getAllOrgs()
renderIds: array[int] = []
for org in orgs {
  dashboards: array[Dashboard] = org.getDashboards()
  for dash in dashboards {
    renderId: int = http.get("http://reporter/render?var-template=${org.name]/${dash.name}").id # 1
    renderIds.push(renderId)
  }
}
awaiters: array[fiber] = []
for id in renderIds {
  awaiters.push(_ => {
    while http.get("http://reporter/status?id=${id}").progress != 100 { # 2
      sleep(1)
    }
  })
}
awaiters.waitForAll()

1) I would like to know if there is a way to receive something like a JSON object to pick just the ID of the report being worked on. 2) I know that /overview and /render?report=$id can show the status, but I would like to programmatically get this value. I could parse the HTML, but once it is done rendering, it seems to instead become a redirect. So I wanted to know if there is a "better" way to do that.

Application call

• type of call [webservice or single-run]: Webservice in asciidoctor image on Kubernetes
• call of reporter [e.g. ruby-grafana-reporter -c myconfig]: See pseudo code above
• call of report creation, if running as webservice [e.g. http://localhost/render?var-template=test]: Above

Template file (if applicable)

• filename: All of them? :)
• relevant part of template file:

  n/a

Configuration file

• created with configuration wizard version [e.g. 0.3.0 or MANUAL]: I don't know, a collegue wrote the deployment - and I just updated it. Chances ae they used the wizard. It is included as a ConfigMap mounted into the container.
• relevant part of configuration file (remember to X-out personal information as e.g. api_key): None specificaly

  ...

Environment (please complete the following information):

• OS [e.g. Windows 10]: Linux, NixOS, amd64
• Grafana version [e.g. 6.3.4]: 11.0.0
• Environment [e.g. source, gem, docker or windows-exe]: docker.io/asciidoctor/docker-asciidoctor:1.72
• Ruby Version, if source or gem environment [e.g. 2.3.7]: As included in the image.
• Reporter version [e.g. 0.3.0]: v0.8.0

Thank you and kind regards!

[divinity666]

Nice use-case you are after! How would you like the REST API to behave? Can you give me some examples?

[senpro-ingwersenk]

Sure!

I'll just make the assumption that the REST API lives in it's own path.

POST /api/report?...
* Same query parameters as the regular /report; but it returns a JSON object instead.
Example:
HTTP/1.1 200 OK
{
  "report_id": 950
}

GET /api/progress?id=$report_id
* Returns a JSON object containing the current status.
Example 1: Report is being worked on
HTTP/1.1 200 OK
{
  "error": { "is_error": false, "message": "" },
  "progress": (float or int representing progress)
}
Example 2: Invalid ID, or report is not being worked on at the moment
HTTP/1.1 404 Not found
{
  "error": { "is_error": true, "message": "Report not found" }
}

GET /api/details?id=$report_id
* Returns the details of a finished report.
Example 1: A finished report
HTTP/1.1 200 OK
{
  "error": { "is_error": false, "message": "" },,
  "url": "<url to fetch the finished report>",
  //... other details that the script might have, that could probably be useful.
}
Example 2: Report not done
HTTP/1.1 412 Not Ready
{
  "error": { "is_error": true, "message": "Report not done yet." },
}
Example 3: Report not found
* Take the 404 example from above; same thing :)

DELETE /api/report?id=$report_id
* Stops/cancels a report generation
* Returns 200 OK on success, 404 Not found on unknown report and otherwise 500 where .error.message describes why.

This should cover most things needed and I am just plotting an example here. :) Still, I hope this gives food for ideas!

With that API I could spawn a Go workgroup/-queue with the report IDs and wait untill they are all done and potentially post-process them (i.e. concat them all into one big PDF covering all dashboards for a certain organization/customer).

You think this's doable?

Kind regards!

[senpro-ingwersenk]

Hello @divinity666 !

Have you had a chance so far to work on this?

Thank you and kind regards. :)

[divinity666]

I investigated several possible solutiont, as well as the integration of an API framework as e.g. grape, but feel that this is too heavy weight for the time being. I guess I will implement it in a similar way, as the existing webservice. To come back to your question: I did not start implementing yet.

[senpro-ingwersenk]

Glad to hear! Don't worry; I just asked back because I started putting together a Go client. Sadly my Ruby-fu is not too good, or I'd see if I could help... sorry. :)

Thanks for your work!

Kind regards

[divinity666]

This should be available with latest release. Please reopen, if something is missing.

[kyouma24]

@senpro-ingwersenk , can you please help me out with grafana here, if you don't mind.

I too have a multi-tenant structure in my grafana (organizations) and I'm using the OSS version.

My config file for reportgenerator consists of a service account's token. Now this service account was created from my main organization. When I switch to other organization, I don't find the service account in there.

I have configured the link for the dashboards in other organizations but the reportgenerator is not able to get the data in the reports because it doesn't have the token from that organization. I've looked up in Grafana's doc and I found that RBAC permissions are available only in Cloud/Enterprise version.

The other solution I think to this is that, I can create mutliple containers and mount the token but then I have too many clients. Can you please let me know how you tackled this situation? Much appreciated!

[senpro-ingwersenk]

Hello @kyouma24 ,

My config file for reportgenerator consists of a service account's token. Now this service account was created from my main organization. When I switch to other organization, I don't find the service account in there.

This is where your solution lies. You will have to use the BasicAuth authentication method in order to log in and iterate through all dashboards Organizations.

See here: https://github.com/senpro-it/grafana-report-generator/blob/master/grafana-client.go#L25-L41

On my end, the account I use is also a "Grafana Server Admin" for what it's worth. It seems that SA Tokens are "per org".

What I did is that I just used that admin account to do... everything. Get all the organizations (which could be filtered by tags), then grab each dashboard's manifest and link them by org.

With that, I just absolutely whallop the reporter with, idk, 80+ requests? I am not exactly there, still fixing the RGR client right now and then moving onward with putting the reports into mails. But basically ... that.

I hope it helped!