search

alexmalins / harlequin-databricks

MIT License
7 stars 0 forks source link

Unable to load catalog #8

Open IvanDerdicDer opened 4 months ago

IvanDerdicDer commented 4 months ago

What is the issue?

Catalog never loads if skip_legacy_indexing if not set in the config, or if skip_legacy_indexing is set to true in config harlequin freezez after loading the catalog. Happens when using either token or oauth authetication. We have Unity Catalog.

Steps to reproduce

I don't know if I can provide a minimal example it happens when i try to connect.

Environment

OS version: Windows 11 Python version: 3.11.9 harlequin-databricks version: 0.3.0 harlequin version: 1.21.0 Installed via: pip

Additional context

alexmalins commented 2 months ago

Hey @IvanDerdicDer Firstly I'm sorry for the slow response - for some reason I missed notifications of these issues so only found out about them now.

Catalog never loads if skip_legacy_indexing if not set in the config, Are you are connecting to a Databricks instance with a large number of tables? My guess is the catalog is indexing, it's just really slow. You can check whether it is trying to index when you load without --skip-legacy-indexing if you look in the Databricks UI Query History, you should be able to see all the SQL commands harlequin-databricks is sending to obtain metadata for every table.

As you can see from the ReadMe, this is a Databricks limitation with legacy metastores (or with Unity Catalog if you load without --skip-legacy-indexing). The only way around it is to wait currently. Maybe there is a way I could improve the code so Harlequin displays the catalogs, schema and tables it has indexed so far in the Data Catalog pane, instead of waiting until everything is indexed to display anything as it does now.

or if skip_legacy_indexing is set to true in config harlequin freezez after loading the catalog. This sounds like a bug... maybe it is Windows related. Could you provide more info to help me debug, e.g. a screen recording? What happens when it freezes? Can you not type anything? Does Harlequin stop responding to any commands, e.g. ctrl+r to refresh the catalog again, ctrl+j to execute a query or ctrl+q to quit?