Support source batches of size > 1 in stream generator

ssec-jhu / dplutils

Distributed(Data) Pipeline Uitilities

BSD 3-Clause "New" or "Revised" License

1 stars 0 forks source link

Support source batches of size > 1 in stream generator #76

Closed amitschang closed 3 weeks ago

amitschang commented 4 months ago

Right now the stream processor can batch input, but it doesn't treat the length of the input at all, so for example if the generator makes dataframes of length 2, and the input task has batch size of 100, the input to that task would be 200 - which is misleading