CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Apache License 2.0

75 stars 16 forks source link

Feature/allow recreation of storage table metadata #69

Closed vavison closed 5 years ago

vavison commented 5 years ago

Description

Currently, once a table is created in the storage layer, there is no way of programmatically updating its metadata. This PR introduces a flag, spark.waimak.storage.updateMetadata, which will force metadata retrieval for all tables and persist the new metadata.

Type of change

Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?

Additional unit tests