search

joocer / cronicl

cronicl: data pipelines.
Apache License 2.0
0 stars 0 forks source link

Calculate a relative priority for stages #12

Open joocer opened 4 years ago

joocer commented 4 years ago

Use as a basis to prioritize stages when the pipeline gets congested;

Prefer stages with lower in:out ratios (reduce the number of records, or create few new records) When there are matching ratios, prefer stages toward the end of the flow

joocer commented 4 years ago

The ratio can be calculated from the input_records and output_records already captured.