feature-request: DIVERSE_REPLICAS to ensure replicas go to different hosts

[navops29]

Consider the below cluster:

cluster carbon_caches
    carbon_ch replication 2
        host1:2003=a
        host1:2103=b
        host1:2203=c
        host2:2003=a
        host2:2103=b
        host2:2203=c
        host3:2003=a
        host3:2103=b
        host3:2203=c

With this set up, the replicas do not necessarily make it to distinct hosts; some times they end up on different instances on the same host. carbon-relay has an option DIVERSE_REPLICAS that guarantees that the replicas end up on distinct hosts. After digging around the docs, I believe this is not currently possible using carbon-c-relay. I'd like to make a feature request to achieve this.

Thanks!

[grobian]

Hmmm. The retrieval code for carbon-c-relay ensures a unique list of servers is returned. See https://github.com/grobian/carbon-c-relay/blob/master/consistent-hash.c#L418. In your scenario you use use multiple caches on a single host, which carbon-c-relay considers all to be unique.

So I see where your problem is. This has never been a problem before, because we simply use a relay on host1, host2 and host3 that any_ofs to any of the tree instances. This change will cause your metrics to get rearranged, so probably not what you're looking for.

Implementing like DIVERSE_REPLICAS would mean adding a check for strcmp(server_ip(ret[j].dest), server_ip(ring->entrylist[t].server)) == 0, which is very expensive and fault prone, so a more expensive check needs to be made like server_cmp() function.