apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.84k stars 4.25k forks source link

[Feature Request]: StateBackedIterable should support lazy observing of element size by extending ElementByteSizeObservableIterable #27283

Closed scwhittle closed 11 months ago

scwhittle commented 1 year ago

What would you like to happen?

As it does not, Iterables.registerByteObserver triggers element iteration and prefetching/reading. Otherwise byte observing would occur lazily during iteration.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

scwhittle commented 11 months ago

.take-issue

scwhittle commented 11 months ago

Renaming to reflect the end goal. I was looking into doing this at the Prefetchables level but it was difficult given overridden methods having colliding types for ElementByteSizeObservableIterable and PrefectIterable.

Planning instead of just fixing up StateBackedIterable and possibly exposing a generic way to wrap an iterable to make it cheap to byte size observe.