search

duckdb / uc_catalog

Proof-of-concept extension combining the delta extension with Unity Catalog
MIT License
51 stars 7 forks source link

Azure support? #7

Open bradlindblad opened 4 months ago

bradlindblad commented 4 months ago

Will there be support for databricks in azure at some point?

gdubya commented 2 months ago

Azure is (possibly) nearly already supported. I tested using the following script:

INSTALL uc_catalog from core_nightly;
INSTALL delta from core;
LOAD delta;
LOAD uc_catalog;
CREATE SECRET (
    TYPE UC,
    TOKEN '${mytoken}',
    ENDPOINT '${my_db_workspace}'
);

CREATE SECRET (
    TYPE AZURE,
    PROVIDER CREDENTIAL_CHAIN,
    CHAIN 'cli',
    ACCOUNT_NAME 'mystorageaccount'
);

ATTACH 'my_catalog' AS my_catalog (TYPE UC_CATALOG);
SHOW ALL TABLES;

select * from my_catalog.my_schema.test_table1;

but encountered https://github.com/duckdb/duckdb_delta/issues/71 when I select from a table.

gdubya commented 1 month ago

This works now. The submodules just need to be bumped to v1.1.0 (CC @samansmink)

glalanne commented 1 month ago

Trying the extension with Azure Databricks using duckdb version v1.1.1 af39bd0dcf.

I can execute the SQL script from above with no errors, but the SHOW ALL TABLES returns 0 rows. Any suggestion to troubleshoot this to see where is the issue?

Using the Databricks CLI returns my tables. Thanks

gdubya commented 1 month ago

@glalanne In a few tests i had to repeat the SHOW ALL TABLES command to get the result. I haven't looked into the cause yet, but perhaps #8 is related?