Integration testing of Pipeline drain in Go SDK

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.78k stars 4.21k forks source link

Integration testing of Pipeline drain in Go SDK #21681

Closed damccorm closed 2 years ago

damccorm commented 2 years ago

Add an integration test for pipeline drain feature BEAM-11106 added in https://github.com/apache/beam/pull/17432.

Imported from Jira BEAM-14406. Original Jira may contain additional context. Reported by: riteshghorse.

damccorm commented 2 years ago

Unable to assign user @riteshghorse. If able, self-assign, otherwise tag @damccorm so that he can assign you. Because of GitHub's spam prevention system, your activity is required to enable assignment in this repo.

riteshghorse commented 2 years ago

@damccorm

damccorm commented 2 years ago

FWIW, this limitation should be addressed by https://github.com/apache/beam/pull/21719

riteshghorse commented 2 years ago

Great, thanks!

lostluck commented 2 years ago

I'll note that we can probably make a special handle for Dataflow jobs so we can programmatically trigger the drain after we have the job confirmed as running.

It wouldn't be able to run via ptest, but that's fine.