search

sparsecode / DaFlow

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
Other
26 stars 13 forks source link

Add Support for the join transformations in the ETL Framework. #26

Open abhioncbr opened 5 years ago

abhioncbr commented 5 years ago

ETL Framework currently supports the basic transformation functions like filter, explode, select. Joining of the two feeds is one of the most common and basic function ETL operations.

abhioncbr commented 5 years ago

Require following changes in ETL Framework

    • ETL SQL Parser for supporting parsing of JOIN conditions,
    • Job Configuration module for handling and parsing of all the required params.
    • In the Core module for handling transformation.