search

databrickslabs / ucx

Automated migrations to Unity Catalog
Other
231 stars 79 forks source link

[BUG]: UCX not setting updating migrated tables in ucx.migration_status table #2537

Closed maruppel closed 1 month ago

maruppel commented 1 month ago

Is there an existing issue for this?

Current Behavior

After running migrate-tables, we can see tables in hive metastore and new Unity Catalog with updated TBL PROPERTIES but these tables are not picked up in the reconciliation workflow and have null dst_catalog/schema/table in the ucx.migration_status table.

Expected Behavior

All tables migrated to new catalog should have non null value for dst_catalog/schema/table in ucx.migration_status tables.

Steps To Reproduce

After running migrate-tables, validate the migration_status table and new catalog schema/tables match. Use custom SQL to determine where a mismatch exists.

Cloud

Azure

Operating System

Windows

Version

latest via Databricks CLI

Relevant log output

No response

maruppel commented 1 month ago

Additionally, it seems when running the migrate-data-reconciliation workflow and errors are encountered it doesn't fail as gracefully it should. This may be where the mismatch is happening, as it's hard to tell what was successful when it fails.

JCZuurmond commented 1 month ago

Hi @maruppel , thank you for opening this issue. Your first comment is covered by the linked PR.

Could you create a separate issue for your second comment? Please add relevant logs to the issue

Additionally, it seems when running the migrate-data-reconciliation workflow and errors are encountered it doesn't fail as gracefully it should. This may be where the mismatch is happening, as it's hard to tell what was successful when it fails.