oliverde8 / php-etl

A small etl coded in php, to extract/transform/load
MIT License
8 stars 4 forks source link

[OPERATION]Add operation to filter data #9

Closed oliverde8 closed 1 year ago

oliverde8 commented 1 year ago

At the moment it's possible to execute an operation on some data transiting in the chain.

For example, we read a CSV file containing customers, we wish to only process customers that has subscribed to the newsletter.

filter-subscribed:
  operation: filter
  options:
     rule: {get : {field: 'is_subscribed'}}

in this example we would use the rule engine to check fields. We could also use the symfony expressions but thos are already built in the rule engine. The rule engine gives more flexibility as it will allow to make checks without failing.


Advance use case

We can use this in with the split operation to make different processes for different data; example:

split-base-on-subscription:
  operation: split
  options:
    execution1: 
      filter-subscribed:
        operation: filter
        options:
           rule: {expression_language: {expression: "rowData.is_subscribed"}}
      custom-transformation-subscribed:
        operation: my-subscribed-operation
        options: []
    execution2: 
      filter-not-subscribed:
        operation: filter
        options:
           rule: {expression_language: {expression: "!rowData.is_subscribed"}}
      custom-transformation-notsubscribed:
        operation: my-notsubscribed-operation
        options: []

!Reminder, any step after the split-base-on-subscription will get all the data both subscribed and unsubscribed in it's state before being transformed by the operations in the split!