databrickslabs / lsql

Lightweight SQL execution wrapper only on top of Databricks SDK
https://pypi.org/project/databricks-labs-lsql/
Other
8 stars 3 forks source link

Fix format breaks widget #238

Closed JCZuurmond closed 1 month ago

JCZuurmond commented 1 month ago

Formatting of a query might break the widget when it contains a CTA. This PR resolves that

Fixes #234

Dashboard from new integration test screenshot. Note that it contains the title on the tile, showing the changes work:

Screenshot 2024-07-25 at 10 24 10
github-actions[bot] commented 1 month ago

❌ 35/36 passed, 1 flaky, 1 failed, 2 skipped, 9m19s total

❌ test_dashboards_creates_dashboard_with_replace_database: databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: (2.567s) ``` databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission CREATE on CATALOG. SQLSTATE: 42501 08:40 DEBUG [databricks.sdk] Loaded from environment 08:40 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 08:40 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 08:40 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 08:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication [gw5] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python 08:40 DEBUG [databricks.sdk] Loaded from environment 08:40 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 08:40 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 08:40 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 08:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication 08:40 DEBUG [databricks.sdk] POST /api/2.0/lakeview/dashboards > { > "display_name": "created_by_lsql_j2wB8JChO2s9z4NQ" > } < 200 OK < { < "create_time": "2024-07-25T08:40:23.343Z", < "dashboard_id": "01ef4a61887511ef9d5d9eb2854c6b85", < "display_name": "created_by_lsql_j2wB8JChO2s9z4NQ", < "etag": "-730165727", < "lifecycle_state": "ACTIVE", < "parent_path": "/Users/4106dc97-a963-48f0-a079-a578238959a6", < "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/created_by_lsql_j2wB8JChO2s9z4NQ.lvdash.json", < "serialized_dashboard": "{\"pages\":[{\"name\":\"3071d8ea\",\"displayName\":\"New Page\"}]}", < "update_time": "2024-07-25T08:40:23.883Z" < } 08:40 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE SCHEMA hive_metastore.lsql_s0dcl WITH DBPROPERTIES (RemoveAfter=2024072509) 08:40 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE SCHEMA hive_metastore.lsql_s0dcl WITH DBPROPERTIES (RemoveAfter=2024072509) 08:40 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/ > { > "format": "JSON_ARRAY", > "statement": "CREATE SCHEMA hive_metastore.lsql_s0dcl WITH DBPROPERTIES (RemoveAfter=2024072509)", > "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID" > } < 200 OK < { < "statement_id": "01ef4a61-88ec-19d8-9315-383691c88797", < "status": { < "error": { < "error_code": "BAD_REQUEST", < "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE on CATA... (20 more bytes)" < }, < "state": "FAILED" < } < } 08:40 DEBUG [databricks.sdk] Loaded from environment 08:40 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 08:40 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 08:40 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 08:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication 08:40 DEBUG [databricks.sdk] POST /api/2.0/lakeview/dashboards > { > "display_name": "created_by_lsql_j2wB8JChO2s9z4NQ" > } < 200 OK < { < "create_time": "2024-07-25T08:40:23.343Z", < "dashboard_id": "01ef4a61887511ef9d5d9eb2854c6b85", < "display_name": "created_by_lsql_j2wB8JChO2s9z4NQ", < "etag": "-730165727", < "lifecycle_state": "ACTIVE", < "parent_path": "/Users/4106dc97-a963-48f0-a079-a578238959a6", < "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/created_by_lsql_j2wB8JChO2s9z4NQ.lvdash.json", < "serialized_dashboard": "{\"pages\":[{\"name\":\"3071d8ea\",\"displayName\":\"New Page\"}]}", < "update_time": "2024-07-25T08:40:23.883Z" < } 08:40 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE SCHEMA hive_metastore.lsql_s0dcl WITH DBPROPERTIES (RemoveAfter=2024072509) 08:40 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE SCHEMA hive_metastore.lsql_s0dcl WITH DBPROPERTIES (RemoveAfter=2024072509) 08:40 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/ > { > "format": "JSON_ARRAY", > "statement": "CREATE SCHEMA hive_metastore.lsql_s0dcl WITH DBPROPERTIES (RemoveAfter=2024072509)", > "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID" > } < 200 OK < { < "statement_id": "01ef4a61-88ec-19d8-9315-383691c88797", < "status": { < "error": { < "error_code": "BAD_REQUEST", < "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE on CATA... (20 more bytes)" < }, < "state": "FAILED" < } < } 08:40 DEBUG [databricks.sdk] DELETE /api/2.0/lakeview/dashboards/01ef4a61887511ef9d5d9eb2854c6b85 < 200 OK < {} [gw5] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python ```

Flaky tests:

Running from acceptance #337

JCZuurmond commented 1 month ago

@nfx : Could you help with the permissions to create schemas from the CI?