vigimite / aqueducts

Framework to build data pipelines declaratively
Apache License 2.0
40 stars 1 forks source link

feat: implement parallel processing of stages #14

Closed vigimite closed 1 week ago

vigimite commented 1 week ago

This PR adds the capability to process stages in parallel by nesting the stages in another vec. Additionally this includes a change to register sources in parallel too.

Usage example:

stages:
  -  - name: is_parallel_a
       query: SELECT * FROM a
     - name: is_parallel_b
       query: SELECT * FROM b
  -  - name: sequential
       query: SELEC x, y FROM a JOIN b

This is a breaking change for the API so the version will be bumped to 0.3.0