ssec-jhu / dplutils

Distributed(Data) Pipeline Uitilities
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Specific handling for zero-length data? #62

Closed amitschang closed 2 months ago

amitschang commented 5 months ago

Zero length batches in the stream graph executor go through the pipeline end will eventually be yielded.

We might want to just discard them (i.e. not add them to the input queue of subsequent tasks or yield), since they don't seem to have much meaning and require book keeping and serde etc.