search

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.37k stars 2.42k forks source link

[SUPPORT] prometheus metrics labels #5326

Open zxding opened 2 years ago

zxding commented 2 years ago

'hoodie.metrics.reporter.metricsname.prefix' = 'hudi_metrics'， Then I got a metrics like hudi_metrics_commit_totalBytesWritten{exported_job="my-hudi-metrics-2", instance="pushgateway", job="pushgateway"} , how to add other label like job_name to make metric like hudi_metrics_commit_totalBytesWritten{exported_job="my-hudi-metrics-2", instance="pushgateway", job="pushgateway" ,job_name = "xxx"} so that I can distinguish metrics from different job

nsivabalan commented 2 years ago

@harsh1231 : Can you chime in here please.

nsivabalan commented 2 years ago

@zxding : guess you are asking for adding arbitrary tags to each metrics right?

zxding commented 2 years ago

yes，any suggestion？

zxding commented 2 years ago

@nsivabalan Thank you for quick reply , I tried to add a property "metricsname.prefix" to make it more flexible for prometheus statistics , but I found that if I did that , the metric lack an tag to distinguish where the metric from , is there any way to archive it ? In release 0.10.0 ,I found the "PushGatewayReporter" class use "DropwizardExports" collector , the DropwizardExports does not use any additional tags , so is it possible to add another implementation of reporter just like "simplecollector" to archive this ? Look forward to your reply.

xushiyan commented 2 years ago

@nsivabalan Thank you for quick reply , I tried to add a property "metricsname.prefix" to make it more flexible for prometheus statistics , but I found that if I did that , the metric lack an tag to distinguish where the metric from , is there any way to archive it ? In release 0.10.0 ,I found the "PushGatewayReporter" class use "DropwizardExports" collector , the DropwizardExports does not use any additional tags , so is it possible to add another implementation of reporter just like "simplecollector" to archive this ? Look forward to your reply.

You're welcome to give a try and make some proof-of-concepts. This is more of a prometheus-specific problem; hopefully someone from the community who is familiar with this can help out here.

nsivabalan commented 2 years ago

@rmahindra123 @harsh1231 : do we have any suggestions on how to go about this.