GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.15k stars 967 forks source link

Add DLQ changes for kafka avro #1770

Open fozzie15 opened 2 months ago

fozzie15 commented 2 months ago

Add DLQ changes for kafka avro to expose the topic, offset, and partition for better visibility into errors.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 0% with 11 lines in your changes missing coverage. Please review.

Project coverage is 42.94%. Comparing base (5de403d) to head (6c1461a). Report is 4 commits behind head on main.

Files Patch % Lines
...teleport/v2/kafka/dlq/BigQueryDeadLetterQueue.java 0.00% 11 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1770 +/- ## ============================================ - Coverage 42.95% 42.94% -0.01% + Complexity 3391 3390 -1 ============================================ Files 815 815 Lines 47635 47643 +8 Branches 5109 5112 +3 ============================================ - Hits 20463 20462 -1 - Misses 25513 25522 +9 Partials 1659 1659 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.77% <ø> (+<0.01%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.40% <ø> (-0.03%)` | :arrow_down: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.17% <ø> (+0.06%)` | :arrow_up: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `71.62% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `83.63% <ø> (ø)` | | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...teleport/v2/kafka/dlq/BigQueryDeadLetterQueue.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770?src=pr&el=tree&filepath=v2%2Fkafka-common%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fkafka%2Fdlq%2FBigQueryDeadLetterQueue.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIva2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvY29tL2dvb2dsZS9jbG91ZC90ZWxlcG9ydC92Mi9rYWZrYS9kbHEvQmlnUXVlcnlEZWFkTGV0dGVyUXVldWUuamF2YQ==) | `0.00% <0.00%> (ø)` | | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1770/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)