grafana / grafana-json-datasource

A data source plugin for loading JSON APIs into Grafana.
Apache License 2.0
269 stars 66 forks source link

Unable to fetch all data from Apis that implement pagination mechanisms #302

Open dpsenner opened 2 years ago

dpsenner commented 2 years ago

There are many api's out there that implement pagination mechanisms. To mention some:

  1. Atlassian Jira
  2. Teamviewer Api

It would be great if the plugin would support these apis better by leveraging the pagination and fetching more data. This is not as easy as it might sound. I can't propose a sound technical spec on how to implement this. Therefore I'm highlighting two concrete examples:

Teamviewer Api endpoint https://webapi.teamviewer.com/api/v1/reports/devices

This api sends an optional next_offset item in the json response that should be sent as http argument offset_id to fetch more data. For example the first response to /reports/devices?from_date=2021-08-01T05:14:00Z json may look like:

{
   ...,
   "next_offset": "eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0="
}

And the following request to fetch more data should now be sent with the parameter /reports/devices?offset_id=eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0=.

A smart guy would be able to base64 decode eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0= to:

{"Start":"2021-08-01T05:14:00+00:00","TableContinuationToken":"8efb4267-2355-4c31-b2bf-48319d32394a"}

Atlassian Jira Api endpoint https://jira.example.com/rest/api/2/search

This api endpoint expects numeric http parameters maxResults and startAt while returning startAt, maxResults, total in the json response. For example requesting ?maxResults=2000&startAt=100 the following is returned:

{
    "startAt": 100,
    "maxResults": 1000,
    "total": 1521,
    ...
}

Note that maxResults is capped at 1000.

vdufloth commented 1 year ago

Hi @dpsenner where you able to connect to Jira using JSON API at all, or did this very issue prevented it from working?

With curl and postman I can fetch the data, but on Grafana it just says: "JSON API: Cannot connect to API"

image

dpsenner commented 1 year ago

@vdufloth Yes, I could fetch the data but ended up with partial results. This is not at all useful and confusing at best. At the moment I can't recall how I managed to get it working.