awslabs / aws-glue-catalog-sync-agent-for-hive

Enables synchronizing metadata changes (Create/Drop table/partition) from Hive Metastore to AWS Glue Data Catalog
Apache License 2.0
33 stars 13 forks source link

doesn't escape nested columns #19

Open Gauravshah opened 2 years ago

Gauravshah commented 2 years ago

For a table with following schema:

 CREATE EXTERNAL TABLE `s3_raw_mongo_snapshots.users`(
`fb_info` struct<_id:string,ext_user_id:string>
)

Athena ends up with :

ParseException line 1:4736 cannot recognize input near '_id:struct' '<' 'Doid' i

It fails to created since glue column cannot start with _ this can be fixed by escaping it

 CREATE EXTERNAL TABLE `s3_raw_mongo_snapshots.users`(
`fb_info` struct<`_id`:string,`ext_user_id`:string>
)