Wrong hashring order

[claudio-benfatto]

Hello, I'm trying to make buckytools work in combination with carbon-c-relay using a jump_fnv1a_ch hash type and replication 1:

the carbon-c-relay configuration is the following:

cluster cluster1
  jump_fnv1a_ch replication 1

  # Select the instances that belong to this cluster based on their index.
  501a1bb8-7860-4b39-8045-470f504ae0a3.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1:2030=cluster1-0 proto tcp

 47063e1b-1a9e-4c23-ac1c-ad187c69f76f.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1:2030=cluster1-2 proto tcp
;

while buckyd is started with the following parameters:

                /var/vcap/jobs/go-carbon/packages/buckytools/bin/buckyd \
                -node 501a1bb8-7860-4b39-8045-470f504ae0a3.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1 \
                -bind 0.0.0.0:4242 \
                -hash jump_fnv1a \
                -prefix /var/vcap/store/go-carbon/whisper/ \
                -replicas 1 \
                501a1bb8-7860-4b39-8045-470f504ae0a3.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1:cluster1-0 47063e1b-1a9e-4c23-ac1c-ad187c69f76f.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1:cluster1-2 

---

the problem is (I think but I'm not sure of it) that the bucky commands are reversing the index of the nodes in the hashring:

graphite-backend/501a1bb8-7860-4b39-8045-470f504ae0a3:/var/vcap/jobs/go-carbon/packages/buckytools/bin# ./bucky servers -h 501a1bb8-7860-4b39-8045-470f504ae0a3.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1:4242 
Buckd daemons are using port: 4242
Hashing algorithm: jump_fnv1a:    0:47063e1b-1a9e-4c23-ac1c-ad187c69f76f.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1     1:501a1bb8-7860-4b39-8045-470f504ae0a3.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1
Number of replicas: 1
Found these servers:
    501a1bb8-7860-4b39-8045-470f504ae0a3.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1
    47063e1b-1a9e-4c23-ac1c-ad187c69f76f.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1

Is cluster healthy: true

and this affects all the other commands as well such as inconsistent and rebalance:

2017/04/09 11:16:14 978 inconsistent metrics found on 501a1bb8-7860-4b39-8045-470f504ae0a3.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1
2017/04/09 11:16:14 956 inconsistent metrics found on 47063e1b-1a9e-4c23-ac1c-ad187c69f76f.graphite-backend.tools.go-graphite-boshrelease.bosh-europe-west1

I was wondering whether I was doing something wrong, especially with the configuration parameters and how could I debug further the cause of these issues.

Many thanks!

[deniszh]

As far as I understand in current implementation hosts should be always sorted, both in carbon-c-relay and bucky. Put 47063e1b-1a9e-4c23-ac1c-ad187c69f76f... host first, and then – 501a1bb8-7860-4b39-8045-470f504ae0a3... - that should work.

[claudio-benfatto]

@deniszh many thanks for your prompt answer.

It seems to me that carbon-c-relay should support sorting via instance labels: https://github.com/grobian/carbon-c-relay/blob/master/carbon-c-relay.md#configuration-syntax

in particular: To influence the ordering, the instance given to the server will be used as sorting key. Without, the order will be as given in the file. It is a good practice to fix the order of the servers with instances such that it is explicit what the right nodes for the jump hash are.

[deniszh]

@claudio-benfatto: Yes, carbon-c-relay should support that, but I didn't test it though. Not sure if buckytools supporting that too.

[claudio-benfatto]

I think I found the source of the issue. Could you please review this PR: https://github.com/jjneely/buckytools/pull/14 when you find some time?

Thanks!

[deniszh]

Sure! But I'm just an ordinary user of a tool, the only @jjneely can merge your change.

[claudio-benfatto]

thanks anyway @deniszh , I really appreciate it :)

[jjneely]

Changes have been merged in.