Closed sivukhin closed 1 month ago
Add few metrics to monitor bottomless
bottomless_s3_processed_frame_ranges
bottomless_local_ready_frame_ranges
bottomless_s3_queue_size
bottomless_local_last_frame_no
bottomless_s3_processing_frame_no
bottomless_local_flush_time
histogram
bottomless_snapshot_upload_time
bottomless_restore_upload_files_time
bottomless_s3_write_time
Example of new metrics part:
# TYPE bottomless_local_ready_frame_ranges counter bottomless_local_ready_frame_ranges{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 1 # TYPE bottomless_s3_processed_frame_ranges counter bottomless_s3_processed_frame_ranges{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 1 # TYPE bottomless_local_last_frame_no gauge bottomless_local_last_frame_no{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 29 # TYPE bottomless_s3_processing_frame_no gauge bottomless_s3_processing_frame_no{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 29 # TYPE bottomless_s3_queue_size gauge bottomless_s3_queue_size{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 0 # TYPE bottomless_local_flush_time summary bottomless_local_flush_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0"} 0.00522327 bottomless_local_flush_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.5"} 0.00522291281325299 bottomless_local_flush_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.9"} 0.00522291281325299 bottomless_local_flush_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.95"} 0.00522291281325299 bottomless_local_flush_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.99"} 0.00522291281325299 bottomless_local_flush_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.999"} 0.00522291281325299 bottomless_local_flush_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="1"} 0.00522327 bottomless_local_flush_time_sum{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 0.00522327 bottomless_local_flush_time_count{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 1 # TYPE bottomless_snapshot_upload_time summary bottomless_snapshot_upload_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0"} 0.000273856 bottomless_snapshot_upload_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.5"} 0.0002738582195644762 bottomless_snapshot_upload_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.9"} 0.0002738582195644762 bottomless_snapshot_upload_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.95"} 0.0002738582195644762 bottomless_snapshot_upload_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.99"} 0.0002738582195644762 bottomless_snapshot_upload_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.999"} 0.0002738582195644762 bottomless_snapshot_upload_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="1"} 0.000273856 bottomless_snapshot_upload_time_sum{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 0.000273856 bottomless_snapshot_upload_time_count{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 1 # TYPE bottomless_restore_upload_files_time summary bottomless_restore_upload_files_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0"} 0.000599889 bottomless_restore_upload_files_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.5"} 0.0005999285280686782 bottomless_restore_upload_files_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.9"} 0.0005999285280686782 bottomless_restore_upload_files_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.95"} 0.0005999285280686782 bottomless_restore_upload_files_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.99"} 0.0005999285280686782 bottomless_restore_upload_files_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.999"} 0.0005999285280686782 bottomless_restore_upload_files_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="1"} 0.000599889 bottomless_restore_upload_files_time_sum{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 0.000599889 bottomless_restore_upload_files_time_count{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 1 # TYPE bottomless_s3_write_time summary bottomless_s3_write_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0"} 0.016363314 bottomless_s3_write_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.5"} 0.016363532981475075 bottomless_s3_write_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.9"} 0.016363532981475075 bottomless_s3_write_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.95"} 0.016363532981475075 bottomless_s3_write_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.99"} 0.016363532981475075 bottomless_s3_write_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="0.999"} 0.016363532981475075 bottomless_s3_write_time{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default",quantile="1"} 0.04838371 bottomless_s3_write_time_sum{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 0.064747024 bottomless_s3_write_time_count{db_name="ns-5d64e223-21a3-4835-9815-9613216d9859:default"} 2
Context
Add few metrics to monitor bottomless
bottomless_s3_processed_frame_ranges
- amount of frame ranges processed by S3 upload processbottomless_local_ready_frame_ranges
- amount of frame ranges ready to upload to S3bottomless_s3_queue_size
- size of the local queue for transferring frames ranges to S3bottomless_local_last_frame_no
- last processed frame number by local processbottomless_s3_processing_frame_no
- last frame number which S3 upload process started to handlebottomless_local_flush_time
- local frames flush timehistogram
bottomless_snapshot_upload_time
- snapshot upload timehistogram
bottomless_restore_upload_files_time
- restore upload files timehistogram
bottomless_s3_write_time
- s3 write timehistogram
Example of new metrics part: