rixed / ramen

A stream processing language and compiler for small-scale monitoring
Other
14 stars 4 forks source link

Reader workers should expose some meta-data on the data source #1377

Open rixed opened 2 years ago

rixed commented 2 years ago

For instance, CSV reader should expose file name, current file offset, current record number, and a "more to come" flag, with which further readers could promptly flush their groups (thus the performance flag). Kafka reader could similarly expose topic name and topic offset as well as message headers.