ytyou / ticktock

TickTockDB is an OpenTSDB-like time series database, with much better performance.
GNU General Public License v3.0
76 stars 9 forks source link

Implementing variables? #69

Open PeterSurda opened 7 months ago

PeterSurda commented 7 months ago

I'm trying to add a variable in grafana from TickTockDB data source, and it doesn't produce any results. I suspect the API to do this isn't implemented in TickTockDB. Do you know more about this? I don't know enough about what's missing and the design in order to implement it.

I use collectd to feed the data, but I put telegraf in the middle so that it can add the necessary tags (collectd's opentsdb output doesn't have tags). I want mainly a variable from the "host" tag, that would be adequate. Then I could select the host for a chart from a drop down menu.

ylin30 commented 7 months ago

@PeterSurda Do you refer to this? I created a custom variable 'HOST' with values (rpi4, raspberrypi, yi-Ideapad), and used it in query tag=value as host=$HOST.

image

image

PeterSurda commented 7 months ago

You got it, except I want the variable values to be generated dynamically. In InfluxDB and QuestDB for example I can specify a query to produce the list:

Grafana documentation says that for OpenTSDB the query is tag_values(host). However when applied on TickTockDB, it produces no results.

PeterSurda commented 7 months ago

Well, what I could do is to have a process, run by a scheduler, that will run a query and use a grafana api call to populate the variable.

ylin30 commented 7 months ago

TT doesn't support APIs (e.g., tag_values(tag)) to retrieve values associated with a tag currently (v.0.12.1). The closest API is suggest?type=tagv&q=&max=1000 which returns all available values for all tags. The results can be filtered by keyword as specified as q=xxxx. Actually when you typed in the value textbox, you can see all unfiltered values for all tags (see the pic below).

To support retrieval of values of a tag, another index between a tag and its values is needed. We will put the functionality in the future list. Hope it is not a dealbreaker for you to use TT.

image

PeterSurda commented 7 months ago

Thank you, I'll find some workaround. Even with this TT seems like the best choice so far (migrating away from rrdtool I also tried graphite carbon, influx, questdb).

This below for example is a quick PoC to produce a list of hosts for a specific chart that have data in the last minute:

curl -s 'http://localhost:6182/api/query?start=1m-ago&m=none:pybitmessagestatus_value.value\{type_instance=networkconnections,host=*\}'|jq -r '[.[].tags.host]|sort|unique|values'
ylin30 commented 7 months ago

Thank you, I'll find some workaround. Even with this TT seems like the best choice so far (migrating away from rrdtool I also tried graphite carbon, influx, questdb).

This below for example is a quick PoC to produce a list of hosts for a specific chart that have data in the last minute:

curl -s 'http://localhost:6182/api/query?start=1m-ago&m=none:pybitmessagestatus_value.value\{type_instance=networkconnections,host=*\}'|jq -r '[.[].tags.host]|sort|unique|values'

That's a smart workaround, even better than building an internal index (which is costly).

PeterSurda commented 7 months ago

Now I just have to figure out how to do that inside grafana rather than in a shell.