electrum
commented
1 year ago This error should be classified better. It should be translated to TrinoException with error code TRANSACTION_CONFLICT. However, the root cause is the concurrency model used by Iceberg.
The table can only be updated by one writer at once. When there are multiple writers, Glue will reject the update for all but the first writer due to the table version being outdated. This is what saves you from losing updates. Iceberg will retry up to 4 times, but with a concurrency of 10, it's likely that one of your workers is always "unlucky" and fails each retry attempt.
Iceberg is not designed for doing many small inserts from independent writers. Some options:
- Coordinate among the workers so that only one is inserting at once.
- Stage all the data somewhere, then insert it at once in one batch.
- Write the data to a queue such as Kafka or Kinesis, then load it from there.
bbarclay
I've run in Kubernetes, as Iceberg, and as Hive table inserts into AWS glue. I'm unsure if this is an AWS Glue problem or a Trino issue. It happens when I run an insert script, at the same time and with concurrency.
MY CODE:
THE ERROR:
TrinoQueryError(type=INTERNAL_ERROR, name=GENERIC_INTERNAL_ERROR, message="Failed to commit to Glue table: order_data.order_test3", query_id=20230328_033606_00016_fe4cq)