Open pequalsnp opened 11 months ago
The Go SDK automatically asserts a Splittable DoFn that returns a ProcessContinuation to be unbounded. But only SplittableDoFns. https://beam.apache.org/documentation/programming-guide/#user-initiated-checkpoint
It's unclear if it's a good idea to allow non-splitable DoFns to be able to self checkpoint. The SDK would need execution side changes to enable it. But from the Beam Portability FnAPI layer, the concepts are tightly related. So at best it would be a convenience layer to things under the hood.
What would you like to happen?
This seems to be supported/explained for Java/Python but not for Golang, for example in the SDF documentation
I think the right thing to do in Golang is implement
IsBounded() bool
, but it would be good to have that in the docs. It is also unclear how to have a regular, non-splittable, do Fn be unbounded if that is even possible. The pubsubio library usesbeam.External
which takes abounded
parameter, but I don't see any other examples of unbounded DoFns in golang.Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components