search

awslabs / aws-glue-catalog-sync-agent-for-hive

Enables synchronizing metadata changes (Create/Drop table/partition) from Hive Metastore to AWS Glue Data Catalog
Apache License 2.0
33 stars 13 forks source link

Make the agent compatible with hive 2.x #7

Closed Aditya-Shah closed 2 years ago

Aditya-Shah commented 3 years ago

Issue

HiveUtils uses method appendSerdeParams from DDLTask in Hive. This method is private in older versions of Hive and leads to following stack:

(RetryingHMSHandler.java:invokeInternal(204)) - java.lang.IllegalAccessError: tried to access method org.apache.hadoop.hive.ql.exec.DDLTask.appendSerdeParams(Ljava/lang/StringBuilder;Ljava/util/Map;)Ljava/lang/StringBuilder; from class com.amazonaws.services.glue.catalog.HiveUtils
    at com.amazonaws.services.glue.catalog.HiveUtils.showCreateTable(HiveUtils.java:208)
    at com.amazonaws.services.glue.catalog.HiveGlueCatalogSyncAgent.onCreateTable(HiveGlueCatalogSyncAgent.java:320)
    at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$12.notify(MetaStoreListenerNotifier.java:75)
    at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:167)
    at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:197)
    at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:235)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1479)
    at

Fix

Add this utility method to the HiveUtils itself to avoid such dependency.

ebirukov commented 2 years ago

Hi @Aditya-Shah, can you upload these changes to s3: // awslabs-code-us-east-1/HiveGlueCatalogSyncAgent ?

IanMeyers commented 2 years ago

Happy to take a pull request that fixes this issue, which can then be pushed out to the distribution on S3.

Aditya-Shah commented 2 years ago

The pull request for this has been merged. We can close the issue. @IanMeyers do we need to have a version update or can newly compiled jars be pushed to S3 as is?

IanMeyers commented 2 years ago

Excellent - I will sort out the version updates and push a new build.

IanMeyers commented 2 years ago

This is now available in version 1.3-SNAPSHOT at s3://awslabs-code-us-east-1/HiveGlueCatalogSyncAgent/HiveGlueCatalogSyncAgent-1.3-SNAPSHOT.jar and s3://awslabs-code-us-east-1/HiveGlueCatalogSyncAgent/HiveGlueCatalogSyncAgent-1.3-SNAPSHOT-complete.jar

Aditya-Shah commented 2 years ago

Thanks @IanMeyers, I am closing the issue.