aws-samples / aws-glue-samples

AWS Glue code samples
MIT No Attribution
1.44k stars 820 forks source link

Add option to put "database_prefix" when import is mode is "from_s3". #66

Open daniloffantinato opened 4 years ago

daniloffantinato commented 4 years ago

Using this approach its possible to import between multiple accounts and add a database prefix.

Hi, I created this change on the import code based on a client engagement on Chile Peru, we needed to have a prefix on databases to import Glue Catalog between AWS accounts. Actually the code is running daily to export from Account A to Account B and its working fine.

Issue #, if available: I founded a issue related to a similar case "https://github.com/aws-samples/aws-glue-samples/issues/43" but I made that based on client needs.

Description of changes: Changed the function "metastore_import_from_s3" and change dataframes "databases", "tables" and "partitions" based on "db_prefix" parameter.

Changed the main and remove database_prefix from not_allowed_options

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.