Refactored etl launch job executor to make more generic & robust

sparsecode / DaFlow

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Other

26 stars 13 forks source link

Refactored etl launch job executor to make more generic & robust #5

Closed abhioncbr closed 5 years ago

abhioncbr commented 6 years ago

Currently, ETL Job Launcher is tightly coupled to validate schema of transformed data & load data will using be validation step results. This should be more generic so that it would use boolean from the job configs to validate transformed data or not.

abhioncbr commented 5 years ago

Several phases of refactoring done which project refactoring, code refactoring. Still work is in progress.