gisaia / ARLAS-proc

Workaround about data ingestion with computing frameworks
Apache License 2.0
4 stars 0 forks source link

Update the transformers wrappers methods #81

Closed laurent-thiebaud-gisaia closed 5 years ago

laurent-thiebaud-gisaia commented 5 years ago

In TransformableDataFrame, wrappers:

Maybe asArlasResampledMotions isn't necessary anymore?

laurent-thiebaud-gisaia commented 5 years ago

By the way, some configuration may also be put in customer specific project with the wrappers, like tempo configurations

laurent-thiebaud-gisaia commented 5 years ago

We stated that DataFrameValidator should be renamed to DataFrameCleaner I prefer to rename it to DataFrameFormatter as it doesn't clean DF, it formats it to a spark-compliant format. As a consequence, the wrapper asArlasFormattedData is made of DataFrameFormatter, WithArlasTimestamp and WithArlasPartition. It may sound strange?

laurent-thiebaud-gisaia commented 5 years ago

Done in https://github.com/gisaia/ARLAS-proc/pull/96