[Q] REMOTE_STORE_MERGE_RESULTS = True doesn't seem to work when schema changes

[jdblack]

Hello all,

We're attempting to convert from a schema that has a 10s default to another that has a 60s default. I performed the following steps:

1. Updated storage-schemas.conf for our graphite-cache cluster so that the default schema has retentions = 60s:14d, with the understanding that pre-existing metrics would retain the old schema
2. Created a new graphite-cluster, also with retentions = 60s:14d by default, with the expectation that all metrics will get the new retetion
3. Set up graphite-web on our relay layer, which has REMOTE_STORE_MERGE_RESULTS = True, to cluster with both the old and new cluster
4. Started marking query reuqests against the relay layer graphite-webs, and observed flapping where each time grafana reloads, we randomly bounce between a long history of metrics, and a much shorter visualization that is clearly just metrics from the new cluster.

Does REMOTE_STORE_MERGE_RESULTS = True work against graphite-webs that have heterogenous clusters? We have quite a bit of metrics and we're hoping that we can avoid shutting caches down to rewrite them with whisper

[deniszh]

Hi @jdblack , Theoretically, that's possible - but nobody implemented that. IIRC carbonapi has ability to merge different retentions - but maybe I'm wrong.

[jdblack]

We haven't been able to move to carbonapi, at least not yet, as carbonapi doesn't support events and does not support some of the graphite-web functions that our developers rely upon.

[deniszh]

Hi @jdblack Theoretically (I didn't test it but should work) you can use carbonapi only as "merger". You can set up carbonapi which will have 2 backends with different retentions, like:

...
backendsv2:
        backends:
          -
            groupName: "graphite-web"
            protocol: "msgpack"
            lbMethod: "broadcast"
            maxTries: 3
            maxBatchSize: 0
            keepAliveInterval: "10s"
            concurrencyLimit: 0
            maxIdleConnsPerHost: 1000
            servers:
                - "http://<ip1>:<port1>?format=msgpack"
                - "http://<ip2>:<port2>?format=msgpack"

and then point another graphite-web to carbonapi as `CLUSTER_SERVERS=[":"]

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.