[Feature Request]: Add support in BQ IO to emit dataset they are accessing for lineage tracking - Githubissues

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.74k stars 4.21k forks source link

[Feature Request]: Add support in BQ IO to emit dataset they are accessing for lineage tracking #31804

Open rohitsinha54 opened 1 month ago

rohitsinha54 commented 1 month ago

What would you like to happen?

BQ IOs should emit dataset they are accessing as lineage information through StringSet metric.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

[X] Component: Python SDK
[X] Component: Java SDK
[X] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

rohitsinha54 commented 1 month ago

.take-issue