Insufficent permission to catalog='system' when using dbt-databricks=1.8.7

VippsMe commented 3 weeks ago

Describe the bug

Issue when using dbt-databricks==1.8.7

We just upgraded from dbt-databricks 1.5 to 1.8.7. This upgrade resulted in an error in the ci job. Downgraded to vs 1.7.17 and then it worked again.
Error message in github action ci job:

[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have USE CATALOG on Catalog 'system'. SQLSTATE: 42501

Steps To Reproduce

See above. Run dbt compile and it will give you the error message. Target is set to a different catalog in databricks with permissions for the user. Somehow, the dbt-databricks adapter is trying to access catalog 'system'. This catalog is not part of our internal code at all.

Expected behavior

See above.

Screenshots and log output

System information

The output of dbt --version:

<output goes here>

The operating system you're using:

The output of python --version:3.10.12

Additional context

Add any other context about the problem here.

benc-db commented 3 weeks ago

This looks like your admin has removed your permission to the system catalog, as every user should have access to this table by default. We can investigate a flag to allow you to reference UC information_schemas scoped by catalog, but we moved to using system catalog as the catalog-scoped information_schema views break if you rename a catalog, so it's kind of a no win situation.

d474b3r9 commented 2 weeks ago

Hello, I encounter the exact same issue even if in my profiles.yml. It works in 1.7.17 but not in 1.8.7 and >

I work on a managed databricks workspace and my company will not allow me rights on the system_catalog.

Do you have a workaround ? Something like set use_system_catalog: false should be great ! ;)

VippsMe commented 1 week ago

a

Its not because of our admin. Access to catalog 'system' has been the exact same all the time. So it is something about dbt-databricks adapter 1.8.XX that is trying to access catalog 'system' (and not our internal repo/code)

benc-db commented 1 week ago

@VippsMe yes, we changed to using system, but we changed because a.) it is more reliable than the individual catalog info_schemas because they are just views of system that can break if the catalog is renamed, and b.) everyone has access to system.information_schema by default. So the reason I'm suggesting that your admin at some point removed your permission, is because you should have access, and if you don't, it's something that an admin can change.

databricks / dbt-databricks