Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Other
193
stars
69
forks
source link
[FEATURE]: Migrate direct filesystem access to UC tables #2021
we have instances of spark.read.format("delta").load("s3a://prefix/...") in the code, though we want to migrate that into spark.table("catalog.schema.table") to follow UC practices.
Unknowns:
do we resolve mounts?
do we resolve dbutils.windgets.get()?
where to store mappings? add a prefix in the table mapping?
what scans all jobs? what determines all direct filesystem accesses? assessment job?
default logic for mapping direct storage prefixes onto databases and catalogs
Is there an existing issue for this?
Problem statement
we have instances of
spark.read.format("delta").load("s3a://prefix/...")in the code, though we want to migrate that intospark.table("catalog.schema.table")to follow UC practices.Unknowns:
dbutils.windgets.get()?Proposed Solution
TBD
Additional Context
No response