search

databrickslabs / ucx

Automated migrations to Unity Catalog
Other
227 stars 80 forks source link

[BUG]: Migration status does not reflect actual table migration status #2904

Open pritishpai opened 1 week ago

pritishpai commented 1 week ago

Is there an existing issue for this?

Current Behavior

For a table that has been migrated to UC the migration status for dst_catalog, dst_schema, dst_table is empty. Upon further debugging, the SHOW TBLPROPERTIES ('upgraded_to') returns empty even when the property exists on the UC table.

One caveat is that the migration has been re-run across several versions and could be a mismatch. For one of the erroneous tables, the destination table was dropped and was successfully migrated in the subsequent run.

Expected Behavior

If a table has been migrated the migration status should reflect the change correctly.

Steps To Reproduce

No response

Cloud

AWS

Operating System

macOS

Version

latest via Databricks CLI

Relevant log output

No response

pritishpai commented 4 days ago

Source table for the views do not have the required 'upgraded_*' properties even after a successful migration. This causes the table migration status to mark the table as not migrated leading to issues in views that depend on this table.