search

go-graphite / carbon-clickhouse

Graphite metrics receiver with ClickHouse as storage
MIT License
186 stars 47 forks source link

Please support carbon tags #10

Closed bzed closed 6 years ago

bzed commented 6 years ago

Since 1.1.0 carbon supports metric tags: http://graphite.readthedocs.io/en/latest/tags.html#

Please support them in carbon-clickhouse! I think storing tags in a carbon compatible way in redis would be the easiest option. Storing them in clickhouse would require graphite-clickhouse to provide a http tag finder service, which should also not be too hard to implement.

I'm happy to help with the coding.

Thanks, Bernd

lomik commented 6 years ago

I'm studying the implementation of tags in graphite-web/carbon. It's very bad.

But I have not yet got to the point of how it works with the CLUSTER_SERVERS.

deniszh commented 6 years ago

But I have not yet got to the point of how it works with the CLUSTER_SERVERS.

All servers in the cluster share same TagDB instance?

lomik commented 6 years ago

All servers in the cluster share same TagDB instance?

graphite-web itself searches for a series of tag and sends a request for each backend to each series? I can't believe this

deniszh commented 6 years ago

@lomik: If we talking about carbon, i.e. writing - task is quite simple: "Carbon will automatically decode the tags, normalize the tag order, and register the series in the tag database." But CLUSTER_SERVERS is part of graphite_web. So, for reading it's doing exactly that - seriesByTag() function calling tagdb.find_series() , which caching results of tagdb._find_series(), which is run query for specific TagDB to get list of metrics by tag.

deniszh commented 6 years ago

I.e. both carbon and graphite-web just share single instance of TagDB service, which can be part of graphite-web, or separate service

lomik commented 6 years ago

Experiment result:

I created 2 series:

:/# echo "hello.world;host=localhost;service=bash 41 1511200882" | nc -q0 graphite 2003
:/# echo "hello.world;host=localhost;service=foobar 42 1511200882" | nc -q0 graphite 2003

then I called seriesByTag from graphite-web with CLUSTER_SERVERS

/render/?width=586&height=308&target=seriesByTag("host=~local*")

Backend received this request

GET /render/?format=pickle&local=1&noCache=1&from=1511115346&until=1511201746&target=hello.world%3Bhost%3Dlocalhost%3Bservice%3Dbash&target=hello.world%3Bhost%3Dlocalhost%3Bservice%3Dfoobar&now=1511201746

Graphite-web found series and fetch them in single request with multiple "target".

Overall need to implement for tag support:

lomik commented 6 years ago

Behavior is changed in graphite-web 1.1.0-rc. seriesByTag filter is passing to CLUSTER_SERVER

/render/?format=pickle&local=1&noCache=1&from=1513155287&until=1513241687&target=seriesByTag%28%27host%3Dlocalhost%27%29&now=1513241687
deniszh commented 6 years ago

Please also note that graphite-clickhouse (and probably go-carbon too) doesn't work with graphite-web 1.1.x because of multiple target params. :( Will try to make a patch.

nikita-b commented 6 years ago

@lomik Hi! 1.0.1 version was released. Do you plan to add support for tags? Thanks!

deniszh commented 6 years ago

Just wanted to note that in current implementation 3rd party tags support can be difficult. Implementing tags store in carbon-clickhouse is relatively easy, but rendering support is much harder. Probably it's easier to write special finder for that or change Graphite-web to not propagate tag calls to backends if HTTP TAGDB is used (then you'll need centralized TagDB storage, but that's a different story).

lomik commented 6 years ago

Please also note that graphite-clickhouse (and probably go-carbon too) doesn't work with graphite-web 1.1.x because of multiple target params. :(

Multiple target params support was added into master branch of graphite-clickhouse

lomik commented 6 years ago

At the moment, tags are supported by carbon-clickhouse (in master) and by graphite-clickhouse (in master):

Not yet supported:

lomik commented 6 years ago

Now tags autocomplete is also supported in graphite-clickhouse master