search

CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Add execute action/function to DataFlow/SparkDataFlow #74

Closed alexjbush closed 5 years ago

alexjbush commented 5 years ago

It might be nice to add an execute action or function to the data flow trait or as an action so a flow can be executed inline:

Waimak.sparkFlow(spark)
    .openCSV(basePath)("csv_1", "csv_2")
    .alias("csv_1", "items")
    .alias("csv_2", "person")
    .writeParquet(baseDest)("items", "person")
    .execute()