Fix BQ deadletter queue for Kafka template

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

https://cloud.google.com/dataflow/docs/guides/templates/provided-templates

Apache License 2.0

1.11k stars 931 forks source link

Fix BQ deadletter queue for Kafka template #1684

Closed AnandInguva closed 2 weeks ago

AnandInguva commented 2 weeks ago

The current BQ dlq fails if the table is already created with empty schema. Change the BQ dlq pipline option template annotation to Text and ask the user for the fully qualified name of the BQ table.

Dippatel98 commented 2 weeks ago

Spotless is failing, can you please fix that.

AnandInguva commented 2 weeks ago

LGTM, but lets wait for the ITs to pass.

Ran the affected test locally and it passed.