search

mrchristine / db-migration

Databricks Migration Tools
Other
43 stars 27 forks source link

Add paging support for large metastore DDL #16

Closed mrchristine closed 4 years ago

mrchristine commented 4 years ago

Need to add paging support for large DDLs.

For files > 1kb, upload to DBFS /tmp/ locations, read the file locally within python, then replay via SparkSQL on the cluster. Files < 1kb, can replay the DDL string directly via the execution context.

mrchristine commented 4 years ago

Supported in the last commit