Hi.
When refreshing table metadata during table loading, there's a long wait time due to retry/backoff on fetching metadata with the underlying file OS. In the case of a permission issue or file not found, the wait time can be 2 mins for a single table.
Checking the underlying code, it's configured to retry 20 times, which isn't a proper amount IMO. There's a fix in Netflix version of Iceberg, where they expose the max-retries as a configurable property, which you can refer to.
EZYZ
commented
2 years ago
Hi. When refreshing table metadata during table loading, there's a long wait time due to retry/backoff on fetching metadata with the underlying file OS. In the case of a permission issue or file not found, the wait time can be 2 mins for a single table.
Checking the underlying code, it's configured to retry 20 times, which isn't a proper amount IMO. There's a fix in Netflix version of Iceberg, where they expose the max-retries as a configurable property, which you can refer to.
Apache version: https://github.com/Netflix/iceberg/blob/master/hive/src/main/java/com/netflix/iceberg/hive/HiveTableOperations.java#L100 and https://github.com/Netflix/iceberg/blob/master/core/src/main/java/com/netflix/iceberg/BaseMetastoreTableOperations.java#L103
Netflix version: https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L207