Retry 429 errors to Dataflow on IT's

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

https://cloud.google.com/dataflow/docs/guides/templates/provided-templates

Apache License 2.0

1.11k stars 931 forks source link

Retry 429 errors to Dataflow on IT's #1690

Closed Polber closed 2 weeks ago

Polber commented 2 weeks ago

Template IT's fail intermittently due to too many calls to Dataflow API per minute (to update pipeline status), so this PR adds retry logic to wait for quota to refresh.