search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

Handle Severe errors: How to process severe errors in Spanner to BigQuery Change stream dataflow #1625

Open pradeepnr opened 3 months ago

pradeepnr commented 3 months ago

Related Template(s)

Spanner_Change_Streams_to_BigQuery

What feature(s) are you requesting?

Hi, we are using Spanner Change streams to BigQuery template to capture cdc.

Occasionally, we have seen errors (not related to template) which are written Retry folder in GCS bucket. After few retrials they are written to severe folder. When the error is fixed we observe the transactions in Retry folder are reprocessed where as the transactions in severe folder are not.

What is the procedure you would recommend to handle transactions in severe folder? Is there any available template we can use? Thanks in Advance.

Regards