It came up in a discussion today that it'd be neat to have stateless domains belowReader nodes that reads could "process" things they extract from Reader state through. Since the domain is stateless, readers could all process this data-flow in parallel using combinations of data they've read without worrying about synchronization with other executors. This would allow us to move certain compute to the readers, which might let us materialize a less verbose Reader further "up" the graph (e.g., above an expensive join).
It came up in a discussion today that it'd be neat to have stateless domains below
Readernodes that reads could "process" things they extract fromReaderstate through. Since the domain is stateless, readers could all process this data-flow in parallel using combinations of data they've read without worrying about synchronization with other executors. This would allow us to move certain compute to the readers, which might let us materialize a less verboseReaderfurther "up" the graph (e.g., above an expensivejoin).