databricks / databricks-sdk-py

Databricks SDK for Python (Beta)
https://databricks-sdk-py.readthedocs.io/
Apache License 2.0
345 stars 114 forks source link

[ISSUE] row_filter name returned from API not parsed correctly #604

Open wsilveira-usp opened 5 months ago

wsilveira-usp commented 5 months ago

Description Databricks /api/2.1/unity-catalog/tables/{full_name} API actually returns unction_name for row_filters and not name as described in the docs.

Reproduction A minimal code sample demonstrating the bug.

Row Filter Function creation:

CREATE FUNCTION some_catalog.some_schema.us_filter(b INT)
RETURN IF(IS_ACCOUNT_GROUP_MEMBER('admin'), true, b=1);

Table creation with Row Filter:

CREATE TABLE some_catalog.some_schema.rectangles_v12(
  a integer,
  b integer,
  c decimal(10, 2),
  `interval` INTERVAL YEAR TO MONTH,
  identity_generated_default BIGINT GENERATED BY DEFAULT AS IDENTITY,
  identity_generated_always BIGINT GENERATED ALWAYS AS IDENTITY ( START WITH 2 INCREMENT BY 2 ),
  identity_default timestamp DEFAULT current_timestamp,
  area INT GENERATED ALWAYS AS (a * b)
) 
USING delta 
LOCATION 'abfss://container@storage_acc.dfs.core.windows.net/some_path'
WITH ROW FILTER some_catalog.some_schema.us_filter ON (b)
TBLPROPERTIES (
  delta.feature.allowColumnDefaults = 'supported',
  custom = 'custom_feature'
);

Fetch table's detail via SDK:

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()

print(w.tables.get("some_catalog.some_schema.rectangles_v12"))

Note the Row filter function name is returned empty

Expected behavior

curl -X GET https://<<WORKSPACE ID>>/api/2.1/unity-catalog/tables/some_catalog.some_schema.rectangles_v12 --header "Authorization: Bearer <<PAT TOKEN>>"

Row Filter function_name returnd not empty

Is it a regression? Don't have past information, but I believe it never worked.

PR Link PR 603

mgyucht commented 5 months ago

Thanks for submitting this and the PR to fix! Unfortunately, this code is autogenerated and will be overwritten by the next SDK release. I'll submit a fix upstream to the underlying API spec so that this is fixed in this SDK and the others as well.

mgyucht commented 5 months ago

The issue was fixed in our upstream API definition. This will be released in this week's SDK release.

wsilveira-usp commented 5 months ago

Ty, @mgyucht !