eBay / HomeObject

Replicated BLOB Store built upon HomeStore
Apache License 2.0
6 stars 15 forks source link

Fill the missing SM-level stats for metrics enhancement #176

Closed yuwmao closed 5 months ago

yuwmao commented 7 months ago

There are 4 SM-level stats we are using.

  1. Space aspects sm total_capacity_bytes sm used_capacity_bytes
  2. Shard aspects sm available_open_shards
    sm open_shards: no value

When testing with emptydir, only sm open_shards doesn't have value. But when testing with pvc, all of these 4 numbers are missing.

Related codes: https://github.com/eBay/HomeObject/blob/7dda682821e6cc1dab78e6630feb3ac99de7487a/src/lib/homestore_backend/hs_homeobject.cpp#L290

JacksonYao287 commented 5 months ago

@yuwmao after the bug fix of CM, we can get the open_shards now. do we still need this issue , or we can close it?

yuwmao commented 5 months ago

As @xiaoxichen mentioned in the morning, when using PVC, get_metrics requests are blocking pg creation. Let's use this issue to track the problem.

JacksonYao287 commented 5 months ago

@yuwmao if that , pleas change to title of this issue, thx!

xiaoxichen commented 5 months ago
        {
            "avail_open_shards": 59997,
            "ctrl_port": 10000,
            "data_port": 10001,
            "id": "ee38ccb2-13d3-4082-8764-8f121ec407ee",
            "ip": "10.12.214.14",
            "loc": {
                "az": "DC1-AZ1",
                "cluster": "FOO",
                "fd": "DC1-AZ1-FD1",
                "host": "tess-node-4nmk6-tess85.stratus.rno.ebay.com",
                "region": "DC1"
            },
            "open_shards": 3,
            "state": 0,
            "total_bytes": 63359447040000,
            "used_bytes": 2587997184
        }

All these 4 fields are reported to CM and available now.

@yuwmao pls double comfirm, if it is good then lets use sepearte SM ticket to track the Metrics