[Q] writing and reading tagged metrics dont appear to line up

[jayninja]

We have a user who is interested in using tagging. But we haven't really been able to get it to work properly with our stack.

We are running go-carbon 0.16.2.
Writes are carbon-c-relay -> go-carbon Reads are graphite-web -> carbonserver

graphite web has the cluster_servers set with the noTags=1 flag.

# cat local_settings.py | grep CLUST
CLUSTER_SERVERS = ["10.0.0.123:8080?noTags=1"]

Without setting this, Grafana would not auto-complete any of the tags. I believe it was due to graphite-web proxying the auto-complete request, and go-carbon returning a 404(or some other non-200) to the auto-complete request.

When writing a tagged metric(with hash-filenames set to false), it writes fine:

# ls -l /graphite/whisper/_tagged/c3f/ec1/some_DOT_metric\;hat\=head.wsp
-rw-r--r-- 1 _graphite _graphite 123089 Dec  1 17:31 '/graphite/whisper/_tagged/c3f/ec1/some_DOT_metric;hat=head.wsp'
#

But when retrieving it, it always came back saying no data.

# curl "http://localhost:8080/render/?format=json&local=1&noCache=1&from=1512146985&until=1669913387&target=some.metric%3Bhat%3Dhead&now=1669913386"
{"metrics":null}
#

In our dev environment, its slow enough that I was able to strace it, and figure out what was going on.

2245258 write(3, "[2022-12-01T17:27:17.594Z] ERROR [carbonserver] open error {\"path\": \"/graphite/whisper/some/metric;hat=head.wsp\", \"error\": \"open /graphite/whisper/some/metric;hat=head.wsp: no such file or directory\"}\n", 201) = 201
2245258 write(7, "[2022-12-01T17:27:17.594Z] ERROR [carbonserver] open error {\"path\": \"/graphite/whisper/some/metric;hat=head.wsp\", \"error\": \"open /graphite/whisper/some/metric;hat=head.wsp: no such file or directory\"}\n", 201) = 201

ngrep carbon-c-relay -> go-carbon

T 10.0.0.122:37994 -> 10.0.0.123:2003 [AP] #1568
some.metric;hat=head 1239 1669912222

So, it looks like carbon-c-relay is passing the metric, and go-carbon is determining it should write it to /graphite/whisper/_tagged/c3f/ec1/some_DOT_metric;hat=head.wsp, but read it from /graphite/whisper/some/metric;hat=head.wsp.

Just to round out the situation, I put a symlink in place, so /graphite/whisper/some/metric;hat=head.wsp -> /graphite/whisper/_tagged/c3f/ec1/some_DOT_metric;hat=head.wsp. The tag then start working.

I'm not entirely sure if I am just doing this wrong, or if this is a bug. But in my instance, if I were to change https://github.com/go-graphite/go-carbon/blob/master/tags/normalize.go#L110 and just have it write to the normal location, it seems like it would line up with the reads going to (/graphite/whisper/some/metric;hat=head.wsp). But im not sure if that breaks other use-cases.

[Civil]

I think the most tested scenario for go-carbon is when hash-filenames = true as that is also somewhat default for upstream carbon. Potentially there might be some bug in handling non-hashed metrics (I'm not familiar with that part of code at all so that's just a wild guess, partially because I use go-carbon with hash-filenames in some test scenarios for carbonapi and that seemed to work fine)

[deniszh]

Hmmm. I was sure that carbonserver don't support tags at all. @Civil, iirc you had some outdated branch with tag support. Or you already merged it? From write path perspective I see no issues - go-carbon doing proper thing here but I think it can be read only through carbonlink, if even possible.

[Civil]

Ah, you are right. I've missed that carbonserver is in use. Indeed, it doesn't support tags. I was working on implementation that would be compatible with graphite-web one, but I need to revive the effort (it actually somewhat worked, I think it works for simple use-cases, but not tested at all for anything complex, plus that was the effort from 2017 with custom storage, and for upstreaming it would require to support whatever upstream carbon supports as well (e.x. sqlite I think and redis with same schemas))

In that case only suggestion would be to either use carbonlink from upstream graphite-web to local graphite-web, that then would fetch files directly and talk with go-carbon using carbonlink. Or as an alternative, you can consider carbon-clickhouse/graphite-clickhouse stack (it's not whisper-compatible, but it supports tags)

[jayninja]

It doesn't seem all that far off from working. After the symlink, the panel in Grafana returned the intended results.

To answer your earlier question:

hash-filenames = false
we write to /graphite/whisper/_tagged/c3f/ec1/some_DOT_metric;hat=head.wsp'

hash-filenames = true we write to /graphite/whisper/_tagged/c3f/ec1/c3fec1656f68b2d5404c2b59754bf6ae42f034c986b62bce3ec23d02297d62b7.wsp

But reads always seem to come from the same place /graphite/whisper/some/metric;hat=head.wsp

carbonserver don't support tags at all.

Ah. bummer, I didnt realize carbonserver itself might not have support.

In that case only suggestion would be to either use carbonlink from upstream graphite-web to local graphite-web, that then would fetch files directly and talk with go-carbon using carbonlink. Or as an alternative, you can consider carbon-clickhouse/graphite-clickhouse stack (it's not whisper-compatible, but it supports tags)

Thanks for the suggestions, I may give this a try, but it seems like tagging might not be worth the architectural changes to the environment to support that. We had some problems with graphite-web hanging when it got a request for a large amount of data on the cache. Hence our graphite-web -> carbon server setup.

I was working on implementation that would be compatible with graphite-web

I will certainly keep my eye out for this. Thanks for the help, and thanks for all the work you guys do @Civil and @deniszh!

[Civil]

Just FYI - I plan to revive the work on tagging support for carbonserver, however I need to have time (and that's an excuse I use for last soon to be 5 years :D )