search

databrickslabs / lsql

Lightweight SQL execution wrapper only on top of Databricks SDK
https://pypi.org/project/databricks-labs-lsql/
Other
8 stars 3 forks source link

Updates to use the Databricks Python sdk 0.30.0 #247

Closed JCZuurmond closed 1 month ago

JCZuurmond commented 1 month ago

These changes contains the update to use the Databricks Python sdk 0.30.0

JCZuurmond commented 1 month ago

@nfx : To cover different versions, we could use tox for testing

github-actions[bot] commented 1 month ago

This PR breaks backwards compatibility for databrickslabs/ucx downstream. See build logs for more details.

Running from downstreams #353

github-actions[bot] commented 1 month ago

❌ 35/36 passed, 1 failed, 2 skipped, 48m31s total

❌ test_dashboards_creates_dashboard_with_replace_database: databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: (2.668s) ``` databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission CREATE on CATALOG. SQLSTATE: 42501 14:12 DEBUG [databricks.sdk] Loaded from environment 14:12 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 14:12 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 14:12 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 14:12 INFO [databricks.sdk] Using Databricks Metadata Service authentication [gw4] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python 14:12 DEBUG [databricks.sdk] Loaded from environment 14:12 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 14:12 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 14:12 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 14:12 INFO [databricks.sdk] Using Databricks Metadata Service authentication 14:12 DEBUG [databricks.sdk] POST /api/2.0/lakeview/dashboards > { > "display_name": "created_by_lsql_H9ym1HBGtjsOFd7B" > } < 200 OK < { < "create_time": "2024-08-13T14:12:30.952Z", < "dashboard_id": "01ef597e141613dabfbdac8de214cbb8", < "display_name": "created_by_lsql_H9ym1HBGtjsOFd7B", < "etag": "1870782005", < "lifecycle_state": "ACTIVE", < "parent_path": "/Users/4106dc97-a963-48f0-a079-a578238959a6", < "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/created_by_lsql_H9ym1HBGtjsOFd7B.lvdash.json", < "serialized_dashboard": "{\"pages\":[{\"name\":\"40f817e7\",\"displayName\":\"New Page\"}]}", < "update_time": "2024-08-13T14:12:31.293Z" < } 14:12 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE SCHEMA hive_metastore.lsql_sqgzg WITH DBPROPERTIES (RemoveAfter=2024081315) 14:12 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE SCHEMA hive_metastore.lsql_sqgzg WITH DBPROPERTIES (RemoveAfter=2024081315) 14:12 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/ > { > "format": "JSON_ARRAY", > "statement": "CREATE SCHEMA hive_metastore.lsql_sqgzg WITH DBPROPERTIES (RemoveAfter=2024081315)", > "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID" > } < 200 OK < { < "statement_id": "01ef597e-1482-1cc5-9ec8-01ee689cf60f", < "status": { < "error": { < "error_code": "BAD_REQUEST", < "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE on CATA... (20 more bytes)" < }, < "state": "FAILED" < } < } 14:12 DEBUG [databricks.sdk] Loaded from environment 14:12 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 14:12 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 14:12 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 14:12 INFO [databricks.sdk] Using Databricks Metadata Service authentication 14:12 DEBUG [databricks.sdk] POST /api/2.0/lakeview/dashboards > { > "display_name": "created_by_lsql_H9ym1HBGtjsOFd7B" > } < 200 OK < { < "create_time": "2024-08-13T14:12:30.952Z", < "dashboard_id": "01ef597e141613dabfbdac8de214cbb8", < "display_name": "created_by_lsql_H9ym1HBGtjsOFd7B", < "etag": "1870782005", < "lifecycle_state": "ACTIVE", < "parent_path": "/Users/4106dc97-a963-48f0-a079-a578238959a6", < "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/created_by_lsql_H9ym1HBGtjsOFd7B.lvdash.json", < "serialized_dashboard": "{\"pages\":[{\"name\":\"40f817e7\",\"displayName\":\"New Page\"}]}", < "update_time": "2024-08-13T14:12:31.293Z" < } 14:12 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE SCHEMA hive_metastore.lsql_sqgzg WITH DBPROPERTIES (RemoveAfter=2024081315) 14:12 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE SCHEMA hive_metastore.lsql_sqgzg WITH DBPROPERTIES (RemoveAfter=2024081315) 14:12 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/ > { > "format": "JSON_ARRAY", > "statement": "CREATE SCHEMA hive_metastore.lsql_sqgzg WITH DBPROPERTIES (RemoveAfter=2024081315)", > "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID" > } < 200 OK < { < "statement_id": "01ef597e-1482-1cc5-9ec8-01ee689cf60f", < "status": { < "error": { < "error_code": "BAD_REQUEST", < "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE on CATA... (20 more bytes)" < }, < "state": "FAILED" < } < } 14:12 DEBUG [databricks.sdk] DELETE /api/2.0/lakeview/dashboards/01ef597e141613dabfbdac8de214cbb8 < 200 OK < {} [gw4] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python ```

Running from acceptance #346

JCZuurmond commented 1 month ago

Integration test failure is unrelated