Closed amitschang closed 4 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 100.00%. Comparing base (
8921afa
) to head (583c41f
).
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Is it necessary to modify our task-specific input/output routines to work with the new OutputBatch format instead of data frames?
@xiangchenjhu, nope - this only affects what gets yielded at the end of the pipeline. So iso
for df in pl.run():
df.to_csv(...)
for example, one would instead:
for batch in pl.run():
batch.data.to_csv(...)
Only for run
, writeto
is back-compatible.
Understood, so this change impacts only the end-of-pipeline usage, while the data flow within each task (both input and output) remains as DataFrames.
To enable partitioning outputs by the task that emitted them. This comes with a breaking change in the interface to executor, since it now emits OutputBatch instead of dataframe, but this seems low risk at the moment, and any users of writeto will remain back-compatible. The OutputBatch structure (as opposed to tuple, for instance) was chosen to enable future metadata additions.
Resolves https://github.com/ssec-jhu/dplutils/issues/61