search

awslabs / aws-glue-catalog-sync-agent-for-hive

Enables synchronizing metadata changes (Create/Drop table/partition) from Hive Metastore to AWS Glue Data Catalog
Apache License 2.0
33 stars 13 forks source link

Huge number of retries of unsuccessful requests in athena #11

Open ebirukov opened 3 years ago

ebirukov commented 3 years ago

One request to the hive metastore to add a partition generates an infinite number of repeated requests to athena. Which seems to prevent all following requests from being run in athena изображение

ebirukov commented 3 years ago

Any unhandled execution error causes execute statement in athena because the completed flag is false https://github.com/awslabs/aws-glue-catalog-sync-agent-for-hive/blob/6dd72172d3a5969a0b559bad2503c76155e1771a/src/main/java/com/amazonaws/services/glue/catalog/HiveGlueCatalogSyncAgent.java#L165-L170

IanMeyers commented 3 years ago

Yes, this is expected as the error may be retryable. What is the base exception causing the create partition statement to fail?

ebirukov commented 3 years ago

Yes, this is expected as the error may be retryable. What is the base exception causing the create partition statement to fail?

In my case,exception causing is table not found, but in a different case, it could be any other and to be generate an infinite number of repeated requests to athena