search

ceph / ceph-medic

find common issues in ceph clusters
MIT License
22 stars 18 forks source link

checks: warn on OSDs with a 0 or below normal CRUSH weight #136

Open alfredodeza opened 5 years ago

alfredodeza commented 5 years ago

Todo: what is normal here? Is 0.0000#### considered a 0?

jeanchlopez commented 5 years ago

Hi Alfredo,

as long as the weight is >= 0 but lower than the other OSDs using the same type of drive it should be flagged. This indicates the cluster PG balance has been altered.

Make sure to check the osd reweight as well as the CRUSH weight. May be in two different iterations to initiate to different warning types.

As the versions move up and with the help of the up-map balancer module this could become more and more seldom across cluster hence the reason to be notified when analyzing a cluster.

alfredodeza commented 5 years ago

Hi Alfredo,

as long as the weight is >= 0 but lower than the other OSDs using the same type of drive it should be flagged. This indicates the cluster PG balance has been altered.

This reads a bit tricky to implement :(

If OSDs are not showing a uniform weight how can I have a baseline that I can compare against? The >= 0 sounds OK, but not sure how to determine if it is lower than the other OSDs using the same type of drive.

Make sure to check the osd reweight as well as the CRUSH weight. May be in two different iterations to initiate to different warning types.

As the versions move up and with the help of the up-map balancer module this could become more and more seldom across cluster hence the reason to be notified when analyzing a cluster.

jeanchlopez commented 5 years ago

Yes I know that's why I really detailed it.

If we could have at least a check on the OSDs that have a 0 CRUSH weight to start that would be a first step.

Then from what we know most customer deploy homogeneous hardware in one node so you need to check all CRUSH weight on a per node basis. Will not span the entire cluster but at least will detect mixed hardware and OSDs that do not have the same CRUSH weight as their peers on a given node as a second step.

Need be I might have some code that I wrote that I could modify and test to do something similar.

jeanchlopez commented 5 years ago

Thsi is an easy way to extarct all the CRUSH weights

ceph osd tree -f json | jq '.nodes[] | {id, crush_weight}'

Note that a lot of customers these days set the osd_crush_initial_weight to 0 to prevent the cluster rebalancing when the add new OSDs and then later on increase the CRUSH weight gradually rather than playing with the OSD reweight ratio.

alfredodeza commented 5 years ago

I have a cluster that has been running for a while, 2 OSDs on the same type of device (HDDs) and their crush_weight values are:

            "crush_weight": 0.010498,
            "crush_weight": 0.005600,

I haven't altered anything other than creating a vanilla Ceph cluster and then launched the OSDs. Would this difference be enough to flag this condition as a warning?

jeanchlopez commented 5 years ago

Yes it would assuming the device class is the same.

JC

While moving. Excuse unintended typos.

On Aug 6, 2019, at 08:06, Alfredo Deza notifications@github.com wrote:

I have a cluster that has been running for a while, 2 OSDs on the same type of device (HDDs) and their crush_weight values are:

        "crush_weight": 0.010498,
        "crush_weight": 0.005600,

I haven't altered anything other than creating a vanilla Ceph cluster and then launched the OSDs. Would this difference be enough to flag this condition as a warning?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.