apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.81k stars 4.23k forks source link

Migrate SnsIO.write to async implementation #21366

Open damccorm opened 2 years ago

damccorm commented 2 years ago

In  BEAM-13203 SnsIO.writeAsync was deprecated to address the risk of data loss in that transform. Unfortunately, it has considerable issues in design and implementation with respect to the beam programming model as well as general async programming:

With BEAM-13563 it will be possible to change SnsIO.write into a more performant async implementation without any breaking changes to the public user facing API making SnsIO.writeAsync obsolete.

Imported from Jira BEAM-13824. Original Jira may contain additional context. Reported by: mosche.

mosche commented 1 year ago

A new async implementation should also use the batch API, see related https://github.com/apache/beam/issues/21429