DataCater / datacater

The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
https://datacater.io
Other
82 stars 4 forks source link

Feature/pipeline spec #16

Closed flippingbits closed 1 year ago

flippingbits commented 1 year ago

This PR introduces a new, more flexible specification for our pipelines.

Instead of fixing the order of filters/transforms and solely operating on single fields of the values of records, we can now chain filters and transforms in an arbitrary manner and apply filters/transforms to both single fields and entire records at a time.

This PR got huge and fixes multiple issues; my apologies - will try to avoid this in the future.

Fixes #14 Fixes #1 Fixes #2