Added support for custom metrics

anuraj381 commented 1 year ago

Details

As of now the connector only exposes standard Kafka Connect MBeans metrics, but we do have lot of operations happening in connector runtime where we are missing observability. This PR adds support for exposing custom metrics from the connector runtime by adding custom JMX MBeans.

Changes

Added MXBean for exporting connector metrics through JMX
Since connector Batch mode load-job-runnables do not have observability, added custom metrics to expose numbers around number of active jobs, storage-blobs etc.
made custom metrics are easily expandable for metrics around various other elements of the connector

Testing

Tested the exported metrics locally with the Jconsole, I am able to see metrics exported from all tasks of the connector.

Note

These custom metrics also facilitates later on adding alerts, for ex. any load-job failures, load-jobs getting triggered in regular intervals etc.
code is expandable for adding more custom metrics from connector sink task

anuraj381 commented 1 year ago

@pbadani @b-goyal @aniketshrimal Hi Guys can you please review this PR

jx2lee commented 3 months ago

Will this PR not be reviewed?

confluentinc / kafka-connect-bigquery