search

go-graphite / buckytools

Go implementation of useful tools for dealing with Graphite's Whisper DBs and Carbon hashing
Other
19 stars 8 forks source link

Cluster in an unhealthy state #28

Closed zerosoul13 closed 2 years ago

zerosoul13 commented 2 years ago

Hello everyone,

Our team has been working on running the go-graphite stack on Kubernetes. We are running the following services:

We are looking to rebalance our cluster go-graphite cluster. We found buckytools but haven't gotten too far since the tool always reports the cluster in an unhealthy state

/ $ /usr/sbin/bucky servers
Buckd daemons are using port: 4242
Hashing algorithm: [carbon: 5 nodes, 100 replicas, 500 ring members go-carbon-0.go-carbon.goal:2004=None go-carbon-1.go-carbon.goal:2004=None go-carbon-2.go-carbon.goal:2004=None go-carbon-3.go-carbon.goal:2004=None go-carbon-4.go-carbon.goal:2004=None]
Number of replicas: 100
Found these servers:
        go-carbon-0.go-carbon.goal
        go-carbon-1.go-carbon.goal
        go-carbon-2.go-carbon.goal
        go-carbon-3.go-carbon.goal
        go-carbon-4.go-carbon.goal

Is cluster healthy: false
2021/10/28 21:29:14 Cluster is inconsistent.

Could you advice how to bring the cluster to a healthy state?

/ $ bucky -version
bucky <sub-command> [options]
Copyright 2015 - 2017 42 Lines, Inc
Original Author: Jack Neely <jjneely@42lines.net>
Version: 0.4.2

        Bucky is a CLI designed to work with large consistent hashing
        Graphite clusters that have the buckyd daemon installed.  Sub-
        commands will allow you to perform high level operations such
        as backups and restores of specific metrics, backfilling, and
        even rebalancing.

        Use the "help" sub-command for available commands.
zerosoul13 commented 2 years ago

To solve the issue, I updated the go-carbon addresses to only use the fqdn and not include the 2004 (Pickle) port