In order to afterwards read files more efficiently using distributed frameworks such as Spark, I added an option to write the parquet file partitioned.
Further, I added an option to add static columns to the final file. This is usefull when the goal is to union tables with the same schema, i.e. add a "year" column for tables storing data from different years.
Added parameters to control chunk and partition size
@tomweber-sas I won't get to it this week, but since you already did the integration into the other IO modules. If you accept these changes, I could add them into the other modules too. I could either do it in this pull request or we open a new one... Or you do it yourself. However you like.
Changes to the sasioiom module:
@tomweber-sas I won't get to it this week, but since you already did the integration into the other IO modules. If you accept these changes, I could add them into the other modules too. I could either do it in this pull request or we open a new one... Or you do it yourself. However you like.
Signed-off-by: Rainer Mensing rainer.mensing@hotmail.de