Closed kkleemola closed 4 years ago
follow up to #476
Dataflow fuses the steps together so adding an extra step doesn't actually help unless we also add something to prevent fusion. There's a few options to prevent this: https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#fusion-optimization
Adding a reshuffle step would be a slightly nicer looking alternative but it is marked as deprecated by beam.
follow up to #476
Dataflow fuses the steps together so adding an extra step doesn't actually help unless we also add something to prevent fusion. There's a few options to prevent this: https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#fusion-optimization
Adding a reshuffle step would be a slightly nicer looking alternative but it is marked as deprecated by beam.