ssec-jhu / dplutils

Distributed(Data) Pipeline Uitilities
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Skip zero-length batches in stream executor #90

Closed amitschang closed 2 months ago

amitschang commented 2 months ago

Zero length data batches risk starting tasks and serializing data unnecessarily, consuming resources for batches that ultimately would not be expected to yield any data.

Resolves https://github.com/ssec-jhu/dplutils/issues/62

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (8e08dbe) to head (e73f30d).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #90 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 11 11 Lines 577 579 +2 ========================================= + Hits 577 579 +2 ``` | [Flag](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/90/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/90/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | `100.00% <100.00%> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.