[flink] Fix source numRecordsIn metrics

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.13k stars 842 forks source link

[flink] Fix source numRecordsIn metrics #3533

Closed tsreaper closed 2 weeks ago

tsreaper commented 2 weeks ago

Purpose

FLIP-27 source has a default implementation of numRecordsIn metrics. However this metrics only count the number of batches Paimon source produces, not the number of records.

This PR fixes the metrics by increasing the counter before emitting data to source output.

Tests

SourceMetricsITCase#testNumRecordsIn

API and Format

No API changes.

Documentation

No new feature.