Tags support

[ipeacocks]

Does carbonapi supports reading tags? I mean this feature https://graphite.readthedocs.io/en/latest/tags.html#carbon

[Civil]

It's not (at this moment), only through graphite-web proxying feature.

[ipeacocks]

Do you have any plans for realizing it?

[Civil]

There are plans, but as I work on this project only at my spare time, I can't tell when it'll be implemented. But I'm open to pull requests here.

I've also heard that one of the forks have this feature implemented, but for some reason they don't want to send it back. As far as I remember it's this fork: https://github.com/errx/carbonapi

[ipeacocks]

@errx Do you have any plans to merge own fork with original repo?

[errx]

Hi. My fork is working only with graphite-clickhouse backend by passing *byTags functions as is.

/tag uri is implemented just as proxy too: https://github.com/errx/carbonapi/blob/merge-CURRENT/tagdb/http.go

So I don't think that my solution is generic enough for a pull request.

[korjjj]

Hi, @errx are you planning to update your fork to more recent version? I'm actually quite interested in tags functionality for graphite-clickhouse backend, but yet failed to build from your branch.

[Civil]

Latest master should now have preliminary tag support, however it's not extensively tested and tag autocomplete queries are not cached as of now.

[Hipska]

Am I correct that you still need graphite-web in order to get it working? Or could some more details given on how to achieve it? Many thanks!

[Civil]

With current master it should work if your backend storage is Clickhouse and you are using graphite-clickhouse.

[Hipska]

Oh, I'm using whisper. That will not work currently?

[Civil]

No, go-carbon do not support returning tagged metrics in a format that can be used by carbonapi.

[b17k0]

With current master it should work if your backend storage is Clickhouse and you are using graphite-clickhouse.

Tags do not work for bundles: graphite-clickhouse (0.9.0) -> carbonapi (0.11.1) -> grafana (6.2.4) Here is what we see when trying to add a variable through the tag - https://i.imgur.com/m9kGBPx.png When we add Grafite to Data Sources and specify the graphite version as 1.1.x, when we open the settings, the version is defined as 1.0.0 (https://i.imgur.com/2OTmQ6s.png) I think the problem is that carbonapi doesn't fully support graphite 1.1.x.

[deniszh]

@beliys : please use master branch or 0.12-rc.0, 0.11 has no tag support.

[Hipska]

Oh, that explains why I didn't got it to work with ClickHouse neither :D

[b17k0]

@deniszh Thanks, the change to the version from the master allowed to change the version of graphite in grafana to 1.1.0. But at the same time, the variable does not receive the value by tag_values(host) in grafana.

[Civil]

@beliys Could you clarify what exactly you are trying to do and what error do you receive?

Ideally with the request information (what was sent and received by carbonapi)

[b17k0]

@Civil I see such errors in carbonapi logs: {"level":"ERROR","timestamp":"2019-06-25T12:54:02.700Z","logger":"access","message":"request failed","data":{"handler":"render","carbonapi_uuid":"54f9f61f-d8ab-4aa1-abc8-98694cf8f0f4","url":"/render","peer_ip":"10.10.10.30","host":"localhost:8081","format":"json","use_cache":true,"targets":["\n.system.load1"],"cache_timeout":60,"runtime":0.000332291,"http_code":400,"reason":"Bad Request\n\nTarget : \n.system.load1\nError : missing argument\nParsed so far : \nCould not parse : \n.system.load1\n","from":1561445642,"until":1561467242,"from_raw":"-6h","until_raw":"now","uri":"/render","from_cache":false}}

{"level":"ERROR","timestamp":"2019-06-25T12:54:35.456Z","logger":"zipper","message":"Setting path cache in non root bg group. somethings off!","type":"broadcastGroup","groupName":"backends","function":"prober"}

[b17k0]

my current config:

listen: "localhost:8081"
concurency: 20
cache:
   type: "mem"
   size_mb: 0
   defaultTimeoutSec: 60
   memcachedServers:
       - "127.0.0.1:1234"
       - "127.0.0.2:1235"
cpus: 0
tz: ""
sendGlobsAsIs: true
maxBatchSize: 100000000
graphite:
    host: ""
    interval: "60s"
    prefix: "carbon.api"
    pattern: "{prefix}.{fqdn}"
idleConnections: 10
pidFile: ""
upstreams:
    buckets: 10
    timeouts:
        global: "20s"
        afterStarted: "2s"
        connect: "200ms"
    concurrencyLimit: 0
    keepAliveInterval: "60s"
    maxIdleConnsPerHost: 100
    backends:
        - "http://127.0.0.1:9090"
        - "http://10.10.33.122:9090"
    carbonsearch:
        backend: "http://127.0.0.1:8070"
        prefix: "virt.v1.*"
    graphite09compat: false
expireDelaySec: 30
logger:
    - logger: ""
      file: "/var/log/carbonapi/carbonapi.log"
      level: "error"
      encoding: "json" 

[Civil]

According to the logs, you were trying to do following query: /render/?target=\n.system.load1

However, currently \ is not a valid symbol for metric, so this request fails with "Bad Request" error.

[b17k0]

In the settings of the dashboard grafana, I specified the request $host.system.load1 (https://i.imgur.com/k4RF082.png) Now I change request to system.load1_host_$host The variable should be parsed by tag: https://i.imgur.com/axt3D1b.png The metrics are collected through a telegraf. In Clickhouse, metrics look like this:

system.load1_host_testnode.domain.com    0.085  1561471980  2019-06-25  1561472029
system.load1_host_testnode.domain.com      0.46 1561472040  2019-06-25  1561472099

I'm not sure that the problem is now on the carbonapi side, but specifically grafana interacts with it.

[Civil]

Ok, and what do you have in the requests to /tags/autoComplete/* urls in the logs?

Also what's the result for /tags/autoComplete/values?format=json&tag=host for both graphite-clickhouse and carbonapi?

[b17k0]

curl -s "http://localhost:8081/tags/autoComplete/*"
Not Found

curl -s "http://localhost:8081/tags/autoComplete/values?format=json&tag=host"
[]

Don't have any errors for this requests for both logs.

How to fix the error

{"level":"ERROR","timestamp":"2019-06-25T12:54:35.456Z","logger":"zipper","message":"Setting path cache in non root bg group. somethings off!","type":"broadcastGroup","groupName":"backends","function":"prober"}

that the logs have become cleaner?

[Civil]

For the query:

curl -s "http://localhost:8081/tags/autoComplete/values?format=json&tag=host"

What's the result if you do it against graphite-clickhouse?

Also there shouldn't be any errors, but some INFO or DEBUG (If enabled) around the request that might give me some idea of what's going on

As about last error - it's fixed in master by https://github.com/go-graphite/carbonapi/commit/4eb4d78384c7b759685cb5e9d2c5922c5084ee0d

[Civil]

I've tried to reproduce your issue on a clean graphite-clickhouse setup, but I actually cannot do that - tag_values in grafana works fine.

Here are the configs I've used: carbonapi.yaml:

listen: "localhost:8081"
concurency: 20
cache:
   type: "mem"
   size_mb: 0
   defaultTimeoutSec: 60
   memcachedServers:
       - "127.0.0.1:1234"
       - "127.0.0.2:1235"
cpus: 0
tz: ""

functionsConfig:
    graphiteWeb: ./graphiteWeb.example.yaml
maxBatchSize: 100
graphite:
    host: ""
    interval: "60s"
    prefix: "carbon.api"
    pattern: "{prefix}.{fqdn}"
idleConnections: 10
pidFile: ""
upstreams:
    buckets: 10

    timeouts:
        find: "2s"
        render: "10s"
        connect: "200ms"
    concurrencyLimitPerServer: 100

    keepAliveInterval: "30s"
    maxIdleConnsPerHost: 100
    backendsv2:
        backends:
          -
            groupName: "clickhouse"
            protocol: "carbonapi_v2_pb"
            lbMethod: "broadcast"
            maxTries: 3
            maxBatchSize: 100
            keepAliveInterval: "10s"
            concurrencyLimit: 100
            maxIdleConnsPerHost: 1000
            backendOptions:
                step: "60"
            timeouts:
                find: "2s"
                render: "50s"
                connect: "200ms"
            servers:
                - "http://127.0.0.1:9090"
    graphite09compat: false
graphTemplates: graphTemplates.example.yaml
expireDelaySec: 10
logger:
    - logger: ""
      file: "stderr"
      level: "debug"
      encoding: "console"
      encodingTime: "iso8601"
      encodingDuration: "seconds"

carbon-clickhouse.toml:

[common]
metric-prefix = "carbon.agents.{host}"
metric-endpoint = "local"
metric-interval = "1m0s"
max-cpu = 1

[logging]
file = "stdout" # /var/log/carbon-clickhouse/carbon-clickhouse.log"
level = "info"

[data]
path = "/Users/civil/graphite-web/storage/carbon-clickhouse/"
chunk-interval = "1s"
chunk-auto-interval = ""
compression = "none"
compression-level = 0

[upload.graphite]
type = "points"
table = "graphite"
threads = 1
url = "http://localhost:8123/"
timeout = "1m0s"
zero-timestamp = false

[upload.graphite_index]
type = "index"
table = "graphite_index"
threads = 1
url = "http://localhost:8123/"
timeout = "1m0s"
cache-ttl = "12h0m0s"

[upload.graphite_tagged]
type = "tagged"
table = "graphite_tagged"
threads = 1
url = "http://localhost:8123/"
timeout = "1m0s"
cache-ttl = "12h0m0s"

[udp]
listen = ":2003"
enabled = true
drop-future = "0s"
drop-past = "0s"

[tcp]
listen = ":2003"
enabled = true
drop-future = "0s"
drop-past = "0s"

[pickle]
listen = ":2004"
enabled = true
drop-future = "0s"
drop-past = "0s"

graphite-clickhouse.conf:

[common]
listen = ":9090"
max-cpu = 1
max-metrics-in-find-answer = 0

[clickhouse]
url = "http://localhost:8123"
extra-prefix = ""
data-table = "graphite"
data-timeout = "1m0s"
rollup-conf = "auto"
index-table = "graphite_index"
index-use-daily = true
index-timeout = "1m"
tagged-table = "graphite_tagged"
tree-table = "graphite_tree"
date-tree-table = ""
date-tree-table-version = 0
tree-timeout = "1m0s"

[prometheus]
external-url = ""
page-title = "Prometheus Time Series Collection and Processing Server"

[carbonlink]
server = ""
threads-per-request = 10
connect-timeout = "50ms"
query-timeout = "50ms"
total-timeout = "500ms"

[[logging]]
logger = ""
file = "stdout"
level = "info"
encoding = "mixed"
encoding-time = "iso8601"
encoding-duration = "seconds"

[b17k0]

For the query: curl -s "http://localhost:8081/tags/autoComplete/values?format=json&tag=host" What's the result if you do it against graphite-clickhouse?

curl -s "http://localhost:8081/tags/autoComplete/values?format=json&tag=host"
[]

carbonapi log:

{"level":"DEBUG","timestamp":"2019-06-26T10:36:14.761Z","logger":"zipper","message":"got some responses","type":"broadcastGroup","groupName":"root","query":"format=json&tag=host","type":"tagValues","backends_count":1,"response_count":1,"have_errors":false}
{"level":"INFO","timestamp":"2019-06-26T10:36:14.761Z","logger":"access","message":"request served","data":{"handler":"tags","carbonapi_uuid":"f9f17c48-0fff-4afa-b0f5-54b9eecccd27","url":"/tags/autoComplete/values","peer_ip":"127.0.0.1","peer_port":"61058","host":"localhost:8081","runtime":0.003249525,"http_code":200,"uri":"/tags/autoComplete/values?format=json&tag=host","from_cache":false}}

graphite-clickhouse log:

{"level":"INFO","timestamp":"2019-06-26T10:44:13.969Z","logger":"query","message":"query","query":"SELECT splitByChar('=', Tag1)[2] AS value FROM graphite_tagged  WHERE (Tag1 LIKE 'host=%') AND (Date >= '2019-06-19') GROUP BY value ORDER BY value LIMIT 10000","request_id":"3289b28cf29207738d10882c05c7db54","time":0.002482491}
{"level":"INFO","timestamp":"2019-06-26T10:44:13.969Z","message":"access","request_id":"3289b28cf29207738d10882c05c7db54","time":0.002689552,"method":"GET","url":"/tags/autoComplete/values?format=json&tag=host","peer":"127.0.0.1:5984","status":200}

I tested on configuration files similar to yours (a little different logger) OS: Debian 9

As about last error - it's fixed in master by 4eb4d78

Thanks, fixed

[Civil]

For the curl - I understand that if you query carbonapi - no results returned.

What I'm interested in is to see what's the results if you query against graphite-clickhouse.

If it's also empty - then there is no issue with carbonapi (except that I treat empty tag as '\n' for some reason), it's just no tagged metrics in clickhouse database.

[b17k0]

Thanks, graphite-clickhouse returns a similar result and probably the problem is not in the carbonapi. Sorry for the time spent.

[Civil]

That's ok :)

If you have any issues with tags support, please open a new issue, as it's not very convenient to track responses in already closed one :)

[b17k0]

I agree, just my first message that the support for tags is not fully implemented, and then there was a discussion of working points. Sorry :)