search

estuary / connectors

Connectors for capturing data from external data sources
Other
38 stars 10 forks source link

materialize-s3-iceberg: tries to re-create glue table and fails #1685

Closed danthelion closed 4 days ago

danthelion commented 1 week ago

Worked for a while, then ran into a loop of trying to re-create the destination table and failing to do so

Materialization: Dani/iceberg-test/materialize-s3-iceberg

Error: 

completeRecovery: store.RestoreCheckpoint: executing apply actions: creating table "dani-ns.Support_Requests": error running iceberg_ctl: Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/pyiceberg/catalog/glue.py", line 335, in _create_glue_table
    self.glue.create_table(DatabaseName=database_name, TableInput=table_input)
  File "/opt/venv/lib/python3.12/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AlreadyExistsException: An error occurred (AlreadyExistsException) when calling the CreateTable operation: Table already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/iceberg-ctl/iceberg_ctl/__main__.py", line 312, in <module>
    run(auto_envvar_prefix="ICEBERG")
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv ...[truncated]
williamhbaker commented 4 days ago

Closed by https://github.com/estuary/connectors/pull/1686