search

s3gw-tech / s3gw

Container able to run on a Kubernetes cluster, providing S3-compatible endpoints to applications.
https://s3gw.tech
Apache License 2.0
142 stars 20 forks source link

Telemetry and Update Responder (Epic) #202

Open l-mb opened 1 year ago

l-mb commented 1 year ago

Rationale

As project sponsors, we would like to understand how often s3gw is deployed in which versions plus some basic facts about the environment. This is important for evaluating the adoption and trajectory of the project and improving the experience for the userbase.

Proposal

This could be implemented similarly to Longhorn or Rancher Telemetry. Perhaps there is an opportunity for code sharing.

We can have the server deployed via our internal IT (proposal to use upgrade-responder.s3gw.io as the endpoint) easily enough.

Then, each s3gw pod issues a periodic (daily) REST POST with the s3gw version as appVersion, and (proposed) the following additional fields in the extraInfo section:

This would be enabled by default, but perhaps we want to make the extraInfo level somewhat configurable so users can opt out of the detailed metrics without depriving us of everything. (off, minimal, standard, detailed)

Links

### Tasks
- [ ] 
- [ ] https://github.com/aquarist-labs/s3gw/issues/443
- [ ] https://github.com/aquarist-labs/s3gw/issues/431
- [ ] https://github.com/aquarist-labs/s3gw/issues/421
- [ ] https://github.com/aquarist-labs/s3gw/issues/423
- [ ] https://github.com/aquarist-labs/s3gw/issues/529
- [ ] https://github.com/aquarist-labs/s3gw/issues/422
l-mb commented 1 year ago

As an example, Rancher Manager just adds the telemetry client/update responder to their container build by default. (It can be disabled via a setting in the UX, but it defaults to on.) I suggest we do the same for s3gw.

This will be critical to show adoption and update rates.

irq0 commented 1 year ago

For reference, this is what the status page exports:

prometheus.txt

Per Counters HTML RGW Status

Perf Counters

Path Description Type Value Type Prio Value
cct.total_workers Total workers gauge int 0 0
cct.unhealthy_workers Unhealthy workers gauge int 0 0
rgw.cache_hit Cache hits counter int 5 0
rgw.cache_miss Cache miss counter int 5 0
rgw.failed_req Aborted requests counter int 5 0
rgw.gc_retire_object GC object retires counter int 5 0
rgw.get Gets counter int 5 0
rgw.get_b Size of gets counter int 5 0
rgw.get_initial_lat Get latency running avg time 5
  • sum: 0
  • count: 0
  • avg: 0
rgw.keystone_token_cache_hit Keystone token cache hits counter int 5 0
rgw.keystone_token_cache_miss Keystone token cache miss counter int 5 0
rgw.lc_abort_mpu Lifecycle abort multipart upload counter int 5 0
rgw.lc_expire_current Lifecycle current expiration counter int 5 0
rgw.lc_expire_dm Lifecycle delete-marker expiration counter int 5 0
rgw.lc_expire_noncurrent Lifecycle non-current expiration counter int 5 0
rgw.lc_transition_current Lifecycle current transition counter int 5 0
rgw.lc_transition_noncurrent Lifecycle non-current transition counter int 5 0
rgw.lua_current_vms Number of Lua VMs currently being executed gauge int 5 0
rgw.lua_script_fail Failed executions of lua scripts counter int 5 0
rgw.lua_script_ok Successfull executions of lua scripts counter int 5 0
rgw.pubsub_event_lost Pubsub events lost counter int 5 0
rgw.pubsub_event_triggered Pubsub events with at least one topic counter int 5 0
rgw.pubsub_events Pubsub events in store gauge int 5 0
rgw.pubsub_missing_conf Pubsub events could not be handled because of missing configuration counter int 5 0
rgw.pubsub_push_failed Pubsub events failed to be pushed to an endpoint counter int 5 0
rgw.pubsub_push_ok Pubsub events pushed to an endpoint counter int 5 0
rgw.pubsub_push_pending Pubsub events pending reply from endpoint gauge int 5 0
rgw.pubsub_store_fail Pubsub events failed to be stored counter int 5 0
rgw.pubsub_store_ok Pubsub events successfully stored counter int 5 0
rgw.put Puts counter int 5 0
rgw.put_b Size of puts counter int 5 0
rgw.put_initial_lat Put latency running avg time 5
  • sum: 0
  • count: 0
  • avg: 0
rgw.qactive Active requests queue gauge int 5 0
rgw.qlen Queue length gauge int 5 0
rgw.req Requests counter int 5 0
rgw_op.abort_multipart counter int 0 0
rgw_op.admin_set_metadata counter int 0 0
rgw_op.bulk_delete counter int 0 0
rgw_op.bulk_upload counter int 0 0
rgw_op.complete_multipart counter int 0 0
rgw_op.config_bucket_meta_search counter int 0 0
rgw_op.copy_obj counter int 0 0
rgw_op.create_bucket counter int 0 0
rgw_op.create_oidc_provider counter int 0 0
rgw_op.create_role counter int 0 0
rgw_op.del_bucket_meta_search counter int 0 0
rgw_op.delete_attrs counter int 0 0
rgw_op.delete_bucket counter int 0 0
rgw_op.delete_bucket_encryption counter int 0 0
rgw_op.delete_bucket_policy counter int 0 0
rgw_op.delete_bucket_public_access_block counter int 0 0
rgw_op.delete_bucket_replication counter int 0 0
rgw_op.delete_bucket_tagging counter int 0 0
rgw_op.delete_cors counter int 0 0
rgw_op.delete_lc counter int 0 0
rgw_op.delete_multi_obj counter int 0 0
rgw_op.delete_obj counter int 0 0
rgw_op.delete_obj_tagging counter int 0 0
rgw_op.delete_oidc_provider counter int 0 0
rgw_op.delete_role counter int 0 0
rgw_op.delete_role_policy counter int 0 0
rgw_op.delete_user_policy counter int 0 0
rgw_op.get_acls counter int 0 0
rgw_op.get_attrs counter int 0 0
rgw_op.get_bucket_encryption counter int 0 0
rgw_op.get_bucket_location counter int 0 0
rgw_op.get_bucket_logging counter int 0 0
rgw_op.get_bucket_meta_search counter int 0 0
rgw_op.get_bucket_obj_lock counter int 0 0
rgw_op.get_bucket_policy counter int 0 0
rgw_op.get_bucket_policy_status counter int 0 0
rgw_op.get_bucket_public_access_block counter int 0 0
rgw_op.get_bucket_replication counter int 0 0
rgw_op.get_bucket_tagging counter int 0 0
rgw_op.get_bucket_versioning counter int 0 0
rgw_op.get_bucket_website counter int 0 0
rgw_op.get_cors counter int 0 0
rgw_op.get_cross_domain_policy counter int 0 0
rgw_op.get_health_check counter int 0 0
rgw_op.get_info counter int 0 0
rgw_op.get_keys counter int 0 0
rgw_op.get_lc counter int 0 0
rgw_op.get_obj counter int 0 0
rgw_op.get_obj_layout counter int 0 0
rgw_op.get_obj_legal_hold counter int 0 0
rgw_op.get_obj_retention counter int 0 0
rgw_op.get_obj_tagging counter int 0 0
rgw_op.get_oidc_provider counter int 0 0
rgw_op.get_request_payment counter int 0 0
rgw_op.get_role counter int 0 0
rgw_op.get_role_policy counter int 0 0
rgw_op.get_user_policy counter int 0 0
rgw_op.init_multipart counter int 0 0
rgw_op.list_bucket counter int 0 0
rgw_op.list_bucket_multiparts counter int 0 0
rgw_op.list_buckets counter int 0 0
rgw_op.list_multipart counter int 0 0
rgw_op.list_oidc_providers counter int 0 0
rgw_op.list_role_policies counter int 0 0
rgw_op.list_role_tags counter int 0 0
rgw_op.list_roles counter int 0 0
rgw_op.list_user_policies counter int 0 0
rgw_op.metadata_search counter int 0 0
rgw_op.modify_role_trust_policy counter int 0 0
rgw_op.options_cors counter int 0 0
rgw_op.period_post counter int 0 0
rgw_op.post_obj counter int 0 0
rgw_op.pubsub_notif_create counter int 0 0
rgw_op.pubsub_notif_delete counter int 0 0
rgw_op.pubsub_notif_list counter int 0 0
rgw_op.pubsub_sub_ack counter int 0 0
rgw_op.pubsub_sub_create counter int 0 0
rgw_op.pubsub_sub_delete counter int 0 0
rgw_op.pubsub_sub_get counter int 0 0
rgw_op.pubsub_sub_pull counter int 0 0
rgw_op.pubsub_topic_create counter int 0 0
rgw_op.pubsub_topic_delete counter int 0 0
rgw_op.pubsub_topic_get counter int 0 0
rgw_op.pubsub_topics_list counter int 0 0
rgw_op.put_acls counter int 0 0
rgw_op.put_bucket_encryption counter int 0 0
rgw_op.put_bucket_obj_lock counter int 0 0
rgw_op.put_bucket_policy counter int 0 0
rgw_op.put_bucket_public_access_block counter int 0 0
rgw_op.put_bucket_replication counter int 0 0
rgw_op.put_bucket_tagging counter int 0 0
rgw_op.put_cors counter int 0 0
rgw_op.put_lc counter int 0 0
rgw_op.put_metadata_account counter int 0 0
rgw_op.put_metadata_bucket counter int 0 0
rgw_op.put_metadata_object counter int 0 0
rgw_op.put_obj counter int 0 0
rgw_op.put_obj_legal_hold counter int 0 0
rgw_op.put_obj_retention counter int 0 0
rgw_op.put_obj_tagging counter int 0 0
rgw_op.put_role_policy counter int 0 0
rgw_op.put_user_policy counter int 0 0
rgw_op.set_attrs counter int 0 0
rgw_op.set_bucket_versioning counter int 0 0
rgw_op.set_bucket_website counter int 0 0
rgw_op.set_request_payment counter int 0 0
rgw_op.set_tempurl counter int 0 0
rgw_op.stat_account counter int 0 0
rgw_op.stat_bucket counter int 0 0
rgw_op.stat_obj counter int 0 0
rgw_op.sts_assume_role counter int 0 0
rgw_op.sts_assume_role_web_identity counter int 0 0
rgw_op.sts_get_session_token counter int 0 0
rgw_op.sync_datalog_notify counter int 0 0
rgw_op.sync_datalog_notify2 counter int 0 0
rgw_op.sync_mdlog_notify counter int 0 0
rgw_op.tag_role counter int 0 0
rgw_op.unknown counter int 0 0
rgw_op.untag_role counter int 0 0
rgw_op.update_role counter int 0 0
rgw_op_svc_time-0x55c02d340500.abort_multipart gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.admin_set_metadata gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.bulk_delete gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.bulk_upload gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.complete_multipart gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.config_bucket_meta_search gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.copy_obj gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.create_bucket gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.create_oidc_provider gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.create_role gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.del_bucket_meta_search gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_attrs gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_bucket gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_bucket_encryption gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_bucket_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_bucket_public_access_block gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_bucket_replication gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_bucket_tagging gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_cors gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_lc gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_multi_obj gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_obj gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_obj_tagging gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_oidc_provider gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_role gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_role_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.delete_user_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_acls gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_attrs gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_encryption gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_location gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_logging gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_meta_search gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_obj_lock gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_policy_status gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_public_access_block gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_replication gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_tagging gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_versioning gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_bucket_website gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_cors gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_cross_domain_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_health_check gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_info gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_keys gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_lc gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_obj gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_obj_layout gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_obj_legal_hold gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_obj_retention gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_obj_tagging gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_oidc_provider gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_request_payment gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_role gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_role_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.get_user_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.init_multipart gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_bucket gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_bucket_multiparts gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_buckets gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_multipart gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_oidc_providers gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_role_policies gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_role_tags gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_roles gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.list_user_policies gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.metadata_search gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.modify_role_trust_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.options_cors gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.period_post gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.post_obj gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_notif_create gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_notif_delete gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_notif_list gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_sub_ack gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_sub_create gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_sub_delete gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_sub_get gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_sub_pull gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_topic_create gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_topic_delete gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_topic_get gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.pubsub_topics_list gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_acls gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_bucket_encryption gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_bucket_obj_lock gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_bucket_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_bucket_public_access_block gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_bucket_replication gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_bucket_tagging gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_cors gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_lc gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_metadata_account gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_metadata_bucket gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_metadata_object gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_obj gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_obj_legal_hold gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_obj_retention gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_obj_tagging gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_role_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.put_user_policy gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.set_attrs gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.set_bucket_versioning gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.set_bucket_website gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.set_request_payment gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.set_tempurl gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.stat_account gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.stat_bucket gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.stat_obj gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.sts_assume_role gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.sts_assume_role_web_identity gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.sts_get_session_token gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.sync_datalog_notify gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.sync_datalog_notify2 gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.sync_mdlog_notify gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.tag_role gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.unknown gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.untag_role gauge time 0 0.000000000s
rgw_op_svc_time-0x55c02d340500.update_role gauge time 0 0.000000000s
rgw_op_svc_time.abort_multipart Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.admin_set_metadata Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.bulk_delete Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.bulk_upload Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.complete_multipart Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.config_bucket_meta_search Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.copy_obj Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.create_bucket Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.create_oidc_provider Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.create_role Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.del_bucket_meta_search Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_attrs Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_bucket Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_bucket_encryption Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_bucket_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_bucket_public_access_block Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_bucket_replication Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_bucket_tagging Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_cors Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_lc Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_multi_obj Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_obj Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_obj_tagging Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_oidc_provider Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_role Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_role_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.delete_user_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_acls Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_attrs Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_encryption Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_location Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_logging Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_meta_search Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_obj_lock Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_policy_status Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_public_access_block Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_replication Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_tagging Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_versioning Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_bucket_website Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_cors Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_cross_domain_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_health_check Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_info Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_keys Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_lc Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_obj Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_obj_layout Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_obj_legal_hold Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_obj_retention Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_obj_tagging Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_oidc_provider Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_request_payment Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_role Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_role_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.get_user_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.init_multipart Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_bucket Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_bucket_multiparts Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_buckets Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_multipart Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_oidc_providers Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_role_policies Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_role_tags Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_roles Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.list_user_policies Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.metadata_search Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.modify_role_trust_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.options_cors Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.period_post Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.post_obj Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_notif_create Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_notif_delete Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_notif_list Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_sub_ack Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_sub_create Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_sub_delete Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_sub_get Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_sub_pull Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_topic_create Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_topic_delete Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_topic_get Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.pubsub_topics_list Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_acls Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_bucket_encryption Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_bucket_obj_lock Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_bucket_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_bucket_public_access_block Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_bucket_replication Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_bucket_tagging Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_cors Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_lc Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_metadata_account Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_metadata_bucket Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_metadata_object Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_obj Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_obj_legal_hold Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_obj_retention Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_obj_tagging Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_role_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.put_user_policy Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.set_attrs Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.set_bucket_versioning Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.set_bucket_website Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.set_request_payment Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.set_tempurl Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.stat_account Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.stat_bucket Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.stat_obj Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.sts_assume_role Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.sts_assume_role_web_identity Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.sts_get_session_token Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.sync_datalog_notify Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.sync_datalog_notify2 Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.sync_mdlog_notify Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.tag_role Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.unknown Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.untag_role Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
rgw_op_svc_time.update_role Histogram of operation service time in µs histogram int 0
<100<1000<1900<3700<7300<14500<28900<57700<115300<230500<460900<921700<1843300<3686500<7372900<14745700<29491300<∞
000000000000000000
simple-throttler.outstanding Outstanding Requests gauge int 0 0
simple-throttler.throttle Requests throttled gauge int 0 0
irq0 commented 1 year ago

https://github.com/aquarist-labs/ceph/pull/133 has code to add a status page exporting what was readily available without spending too much time :)

Output:

$ curl localhost:9090/telemetry
{
    "appVersion": "18.0.0-3118-ge21d2ab4b05",
    "extraInfo": {
        "ceph_version": "ceph version 18.0.0-3118-ge21d2ab4b05 (e21d2ab) reef (dev)",
        "ceph_version_short": "18.0.0-3118-ge21d2ab4b05",
        "ceph_release": "reef",
        "os": "Linux",
        "kernel_version": "6.1.7-1-default",
        "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c)",
        "arch": "x86_64",
        "mem_total_kb": "32486064",
        "mem_swap_kb": "32485372",
        "cpu": "Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz",
        "cgroup_mem_limit": "0",
        "sfs_avail_kb": 4315241,
        "sfs_total_kb": 300933564,
        "sfs_used_kb": 257781152,
        "sfs_fs": "xfs"
    }
}

We could use this with something like curl localhost:9090/telemetry | curl https://??.s3gw.io in a container startup to get a simple first version out.

l-mb commented 1 year ago

aquarist-labs/ceph#133 has code to add a status page exporting what was readily available without spending too much time :)

Nice! Good start. I think we may want to trim this a little to reduce redundancy.

"extraInfo": {
    "ceph_version": "ceph version 18.0.0-3118-ge21d2ab4b05 (e21d2ab) reef (dev)",
    "ceph_version_short": "18.0.0-3118-ge21d2ab4b05",
    "ceph_release": "reef",

We can drop these three, we already push the appVersion

    "os": "Linux",

Maybe drop? Would this be different on, say, WSL?

    "mem_total_kb": "32486064",
    "mem_swap_kb": "32485372",

Can we add how much memory our processes are using?

    "cpu": "Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz",
    "cgroup_mem_limit": "0",
    "sfs_avail_kb": 4315241,
    "sfs_total_kb": 300933564,
    "sfs_used_kb": 257781152,
    "sfs_fs": "xfs"

Can we add object/bucket counts (plus logical size)?

k8s version if we could, and the PVC type, but those can be add-ins. Can we tell whether we've been started by systemd or k8s?

We could use this with something like curl localhost:9090/telemetry | curl https://??.s3gw.io in a container startup to get a simple first version out.

This won't be enough, since we need to run it periodically (as instructed by the telemetry server). I don't think this is a short cut worth taking, we should directly go for a thread or some such.

irq0 commented 1 year ago

aquarist-labs/ceph#133 has code to add a status page exporting what was readily available without spending too much time :)

Nice! Good start. I think we may want to trim this a little to reduce redundancy.

"extraInfo": {
    "ceph_version": "ceph version 18.0.0-3118-ge21d2ab4b05 (e21d2ab) reef (dev)",
    "ceph_version_short": "18.0.0-3118-ge21d2ab4b05",
    "ceph_release": "reef",

We can drop these three, we already push the appVersion

Agree on ceph_version_short (== appVersion) and ceph_release. ceph_version (== radosgw --version) may be worth keeping. In some cases it has extra info like the build type and the git version.

    "os": "Linux",

Maybe drop? Would this be different on, say, WSL?

WSL: no idea.

The function that returns that value has provisions for Windows (w/o WSL), *BSD and macs. Might be cool to see the first non Linux build.

    "mem_total_kb": "32486064",
    "mem_swap_kb": "32485372",

Can we add how much memory our processes are using?

Couldn't find anything in Ceph's utils :(

Does PSS collected from smaps make sense (e.g awk '/Pss:/ { pss+=$2 } END { print pss }' /proc/$(pgrep radosgw)/smaps) ?

    "cpu": "Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz",
    "cgroup_mem_limit": "0",
    "sfs_avail_kb": 4315241,
    "sfs_total_kb": 300933564,
    "sfs_used_kb": 257781152,
    "sfs_fs": "xfs"

Can we add object/bucket counts (plus logical size)?

:+1: 

select count(*) as sfs_bucket_count from buckets;
select count(*) as sfs_object_count from objects;
select sum(size)/1024/1024 as sfs_object_size_total_mb from versioned_objects where object_state = /* ObjectState::COMITTED */;

Any other queries to the db?

k8s version if we could, and the PVC type, but those can be add-ins. Can we tell whether we've been started by systemd or k8s?

We can guess based on environment vars? Paging @m-ildefons

What we can easily have is: pod_name, container_name, container_image, container_hostname, pod_namespace. Ceph has the guesswork code already done. container_image is in the PR, the rest feel a bit too private, no?

We could use this with something like curl localhost:9090/telemetry | curl https://??.s3gw.io in a container startup to get a simple first version out.

This won't be enough, since we need to run it periodically (as instructed by the telemetry server). I don't think this is a short cut worth taking, we should directly go for a thread or some such.

Oh yes, just read it. The telemetry server expects hourly pings.

m-ildefons commented 1 year ago

A container in side a K8s environment will have a number of things injected by K8s by default. Though different K8s distributions like K3s may differ on this slightly, there's a good chance that most of it will be present in all K8s distributions. What springs to mind as most indicative of a container running in K8s are e.g.:

The presence of either (or all) of these would be a strong indicator that the container is running in K8s. But to be completely sure, the environment variables can be pieced together to the K8s API endpoint and that can be queried for the K8s version, e.g.:

db-0:/ # curl -k https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/version
{
  "major": "1",
  "minor": "24",
  "gitVersion": "v1.24.8",
  "gitCommit": "fdc77503e954d1ee641c0e350481f7528e8d068b",
  "gitTreeState": "clean",
  "buildDate": "2022-11-09T13:31:40Z",
  "goVersion": "go1.18.8",
  "compiler": "gc",
  "platform": "linux/amd64"
}
l-mb commented 1 year ago

The k8s version / gitVersion / Commit / platform is probably the most relevant to understand what distro we're on.

Might make sense to shift the discussion about the exact metrics to a separate issue, I'd recommend? Thanks!

PhanLe1010 commented 1 year ago

I think it might be helpful to call out that longhorn/upgrade-responder is planning to support InfuxDB field (can be number) in addition to currently supported tags (string only). This would help with data that are not string like "sfs_total_kb": 300933564,. Ref: https://github.com/longhorn/upgrade-responder/pull/48

This will be release soon

jecluis commented 7 months ago

This will need to be reassessed. I believe telemetry is valuable, but likely no longer targeting longhorn/rancher's endpoints.