Can this be docked with Grafana？

[nanquanyuhao]

Hello...

With your component radosgw_usage_exporter , I have pull data with Prometheus. But I wanna view data with "grafana + Prometheus", do you have any dashboard mix this exporter? Can you help me ?

Thanks!

By the way , why do I only see these data?

[blemmenes]

Hello,

You can certainly use Grafana to view these metrics, I have not however built any grafana dashboards specific to the metrics produced by this exporter. It is on my list of things to do, but not something I will get to in the next week or two.

Are those the extent of the metrics you get back? Can you provide info on what ceph version this is running against?

Also, does the user you are using with this module have both usage= read and buckets=read capabilities?

Thanks, Berant

[nanquanyuhao]

@blemmenes Thanks for your response...

That picture content is indeed all the data... And I use the docker image ceph/demo:tag-build-stable-2.2.11-jewel-centos-7 from dockehub. The account I used info like this and I think it has usage= read and buckets=read capabilities:

{
    "user_id": "admin",
    "display_name": "Administrator",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "admin",
            "access_key": "6H23GNHCC1QTMSRJUSHX",
            "secret_key": "arKl2acDMKnrUsBDQGJVxO6U28m8txXzxEdw32lN"
        }
    ],
    "swift_keys": [],
    "caps": [
        {
            "type": "buckets",
            "perm": "*"
        },
        {
            "type": "metadata",
            "perm": "*"
        },
        {
            "type": "usage",
            "perm": "*"
        },
        {
            "type": "users",
            "perm": "*"
        },
        {
            "type": "zone",
            "perm": "*"
        }
    ],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

But I hadn't add rgw enable usage log = true to ceph.conf, is that the reason?

Next I will install your component radosgw_usage_exporter directly on my ceph gluster in VM...

By the way , My English is very poor, Thanks for your help again...

[blemmenes]

Yeah, if you haven't enabled the usage log you won't get any of that data. Let me know if you have any issues once you enable that.

No worries on your English.

Thanks, Berant

[nanquanyuhao]

@blemmenes But as you saw, I have some data like this only this... Is it all?

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 179810304.0
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 15400960.0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1524539063.39
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 101.48
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 65536.0
# HELP radosgw_usage_successful_ops_total Number of successful operations
# TYPE radosgw_usage_successful_ops_total counter
# HELP radosgw_usage_bucket_total Bucket used bytes
# TYPE radosgw_usage_bucket_total gauge
radosgw_usage_bucket_total{bucket="test10",owner="1_1_1524067231262",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test2",owner="1_1_1523696628085",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="my-new-bucket",owner="testuser",zonegroup="0"} 40960.0
radosgw_usage_bucket_total{bucket="test7",owner="1_1_1524066529235",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test11",owner="1_1_1524067371106",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test4",owner="1_1_1523697318438",zonegroup="0"} 36864.0
radosgw_usage_bucket_total{bucket="tset12",owner="1_1_1524067397231",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="lis001",owner="1_1_1524651275973",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test9",owner="1_1_1524067064272",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test",owner="1_1_1523692816592",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="myfirstbucket",owner="1_1_1523619384379",zonegroup="0"} 40960.0
radosgw_usage_bucket_total{bucket="thsd11",owner="3_3_1524362118250",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test8",owner="1_1_1524066830287",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test6",owner="1_1_1524042799129",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test5",owner="1_1_1523757594115",zonegroup="0"} 0.0
radosgw_usage_bucket_total{bucket="test3",owner="1_1_1523696805278",zonegroup="0"} 36864.0
radosgw_usage_bucket_total{bucket="2test",owner="1_1_1524124857513",zonegroup="0"} 0.0
# HELP radosgw_usage_received_bytes_total Bytes received by the RADOSGW
# TYPE radosgw_usage_received_bytes_total counter
# HELP radosgw_usage_sent_bytes_total Bytes sent by the RADOSGW
# TYPE radosgw_usage_sent_bytes_total counter
# HELP radosgw_usage_ops_total Number of operations
# TYPE radosgw_usage_ops_total counter

[blemmenes]

@nanquanyuhao This is after you enabled RGW usage logging and generated some traffic?

I'll work on firing up a cluster using the demo container and see what I get.

Thanks, Berant

[nanquanyuhao]

@blemmenes No, this is the previous data...

[blemmenes]

@nanquanyuhao So have you configured 'rgw enable usage log = true'?

[nanquanyuhao]

@blemmenes As you say, my data like this when I had enabled the usage log...

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 179806208.0
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 15282176.0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1525112440.58
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.43000000000000005
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 65536.0
# HELP radosgw_usage_successful_ops_total Number of successful operations
# TYPE radosgw_usage_successful_ops_total counter
radosgw_usage_successful_ops_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 1.0
radosgw_usage_successful_ops_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 3.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 2.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 3.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 2.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 4.0
radosgw_usage_successful_ops_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 4.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 1.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 1.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 1.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 2.0
radosgw_usage_successful_ops_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 2.0
# HELP radosgw_usage_bucket_total Bucket used bytes
# TYPE radosgw_usage_bucket_total gauge
radosgw_usage_bucket_total{bucket="my-new-bucket",owner="testuser",zonegroup="83d27dfb-c465-4762-9f7e-3e326a30764e"} 40960.0
radosgw_usage_bucket_total{bucket="your-new-bucket",owner="testuser",zonegroup="83d27dfb-c465-4762-9f7e-3e326a30764e"} 40960.0
# HELP radosgw_usage_received_bytes_total Bytes received by the RADOSGW
# TYPE radosgw_usage_received_bytes_total counter
radosgw_usage_received_bytes_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 0.0
radosgw_usage_received_bytes_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 73094.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_received_bytes_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 36547.0
# HELP radosgw_usage_sent_bytes_total Bytes sent by the RADOSGW
# TYPE radosgw_usage_sent_bytes_total counter
radosgw_usage_sent_bytes_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 214.0
radosgw_usage_sent_bytes_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 957.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 36559.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 1634.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 12.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 819.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 0.0
radosgw_usage_sent_bytes_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 0.0
# HELP radosgw_usage_ops_total Number of operations
# TYPE radosgw_usage_ops_total counter
radosgw_usage_ops_total{bucket="bucket_root",category="list_buckets",owner="anonymous"} 1.0
radosgw_usage_ops_total{bucket="bucket_root",category="list_buckets",owner="testuser"} 3.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="create_bucket",owner="testuser"} 2.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="get_obj",owner="testuser"} 3.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="list_bucket",owner="testuser"} 2.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="put_acls",owner="testuser"} 4.0
radosgw_usage_ops_total{bucket="my-new-bucket",category="put_obj",owner="testuser"} 4.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="create_bucket",owner="testuser"} 1.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="get_obj",owner="testuser"} 1.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="list_bucket",owner="testuser"} 1.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="put_acls",owner="testuser"} 2.0
radosgw_usage_ops_total{bucket="your-new-bucket",category="put_obj",owner="testuser"} 2.0

I still hope you will built some grafana dashboards specific to the metrics produced by this exporter as quickly as possible， thank you very much...

[blemmenes]

I apologize if I'm missing your issue but is this data not what you would expect? Now that you have enabled usage logging the above output looks normal to me.

[nanquanyuhao]

@blemmenes This data is what I need... If enable the usage log , this data is what I need... 3Q..

[blemmenes]

@nanquanyuhao I'm afraid I'm misunderstanding what you are missing. This exporter only produces 5 different metrics and in your example above I show data from all 5.

Can you please show examples of what you expect to see that you do not?

Thanks, Berant

[nanquanyuhao]

@blemmenes I am meaning that when I enable the usage log, this exporter can produces all the 5 different metrics... Before I configuring it, I can't get the all 5 metrics but only radosgw_usage_bucket_total...

Very thanks...

[blemmenes]

Ah, in that case it is expected behavior. The exporter cannot show the usage metrics if the ceph RADOSGWs are not tracking and recording the usage information.

It is a requirement of this exporter (and any other tool that wants to get usage information out of the admin ops API) that rgw enable usage log = true is enabled.

Thanks, Berant

[k0ste]

Since Luminous 12.2.5 mgr was get rgw metrics support (https://github.com/ceph/ceph/pull/21303) we can mix usage of this exporter and mgr/prometheus. With metadata we can get metrics of each rgw instance.

[nanquanyuhao]

@k0ste Thinks... In fact, what I need is the usage of bucket for user... I have finish it like this:

Thinks ant way...

[k0ste]

@nanquanyuhao bucket usage or "how much eat each user" is easy:

[blemmenes]

@k0ste I've not really done any investigation of what the manager prometheus instrumentation exposes. I used ceph-nano to spin up a 12.2.5 cluster and this is what I got after creating a bucket and uploading one file to it.

# HELP ceph_rgw_qlen Queue length      
# TYPE ceph_rgw_qlen gauge             
ceph_rgw_qlen{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0           
# HELP ceph_rgw_req Requests           
# TYPE ceph_rgw_req counter            
ceph_rgw_req{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 5.0            
# HELP ceph_rgw_cache_miss Cache miss  
# TYPE ceph_rgw_cache_miss counter     
ceph_rgw_cache_miss{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 2.0     
ceph_objecter_op_rmw{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0    
# HELP ceph_rgw_put Puts               
# TYPE ceph_rgw_put counter            
ceph_rgw_put{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 1.0            
ceph_pool_metadata{pool_id="4",name="default.rgw.log"} 1.0                     
ceph_pool_metadata{pool_id="3",name="default.rgw.meta"} 1.0                    
ceph_pool_metadata{pool_id="5",name="default.rgw.buckets.index"} 1.0           
ceph_pool_metadata{pool_id="1",name=".rgw.root"} 1.0                           
ceph_pool_metadata{pool_id="2",name="default.rgw.control"} 1.0                 
ceph_pool_metadata{pool_id="6",name="default.rgw.buckets.data"} 1.0            
ceph_objecter_op_active{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0 
# HELP ceph_rgw_keystone_token_cache_hit Keystone token cache hits             
# TYPE ceph_rgw_keystone_token_cache_hit counter                               
ceph_rgw_keystone_token_cache_hit{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0                                                                      
# HELP ceph_rgw_metadata RGW Metadata  
# TYPE ceph_rgw_metadata untyped       
ceph_rgw_metadata{id="ceph-nano-luminous-faa32aebf00b",hostname="ceph-nano-luminous-faa32aebf00b",ceph_version="ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)"} 1.0
# HELP ceph_rgw_qactive Active requests queue                                  
# TYPE ceph_rgw_qactive gauge          
ceph_rgw_qactive{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0        
# HELP ceph_rgw_get_b Size of gets     
# TYPE ceph_rgw_get_b counter          
ceph_rgw_get_b{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0          
# HELP ceph_rgw_failed_req Aborted requests                                    
# TYPE ceph_rgw_failed_req counter     
ceph_rgw_failed_req{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 1.0     
# HELP ceph_rgw_keystone_token_cache_miss Keystone token cache miss            
# TYPE ceph_rgw_keystone_token_cache_miss counter                              
ceph_rgw_keystone_token_cache_miss{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0                                                                     
# HELP ceph_rgw_get Gets               
# TYPE ceph_rgw_get counter            
ceph_rgw_get{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 0.0            
# HELP ceph_rgw_cache_hit Cache hits   
# TYPE ceph_rgw_cache_hit counter      
ceph_rgw_cache_hit{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 12.0     
ceph_objecter_op_w{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 1692.0   
ceph_objecter_op_r{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 337.0    
# HELP ceph_rgw_put_b Size of puts     
# TYPE ceph_rgw_put_b counter          
ceph_rgw_put_b{ceph_daemon="rgw.ceph-nano-luminous-faa32aebf00b"} 5469.0     

I don't see any overlap with the metrics we are getting via the usage/bucket API though I could be missing something...

[k0ste]

No really overlap, because radosgw_usage_exporter is focused on bucket usage/ops, mgr/prometheus is focused on rgw instance requests/data transmission. So sum of ops/bytes per bucket we have is the one total metric of mgr, per instance.

[icoolchn]

Since Luminous 12.2.5 mgr was get rgw metrics support (ceph/ceph#21303) we can mix usage of this exporter and mgr/prometheus. With metadata we can get metrics of each rgw instance.

hello, give me the mgr rgw grafana dashaboard ,thanks

[symbatkarzhassov]

https://github.com/ceph/ceph/tree/master/monitoring/grafana/dashboards