Release v0.7.3

[nfx]

• Added publish flag to Dashboards.create_dashboard (#233). In this release, we have added a publish flag to the Dashboards.create_dashboard method, allowing users to publish the dashboard upon creation, thereby resolving issue #219. This flag is included in the labs.yml file with a description of its functionality. Additionally, the no-open flag's description has been updated to specify that it prevents the dashboard from opening in the browser after creation. The create_dashboard function in the cli.py and dashboards.py files has been updated to include the new publish flag, allowing for more flexibility in how users create and manage their dashboards. The Dashboards.create_dashboard method now calls the WorkspaceClient.lakeview.publish method when the publish flag is set to True, which publishes the created dashboard. This behavior is covered in the updated tests for the method.
• Fixed boolean cli flags (#235). In this release, we have improved the handling of command-line interface (CLI) flags in the databricks labs command. Specifically, we have addressed the limitation that pure boolean flags are not supported. Now, when using boolean flags, the user will be prompted to confirm with a y or 'yes'. We have modified the create_dashboard command to accept string inputs for the publish and no_open flags, which are then converted to boolean values for internal use. Additionally, we have introduced a new open-browser command, which will open the dashboard in the browser after creating when set to y or 'yes'. These changes have been tested manually to ensure correct behavior. This improvement provides a more flexible input experience and better handling of boolean flags in the CLI command for software engineers using the open-source library.
• Fixed format breaks widget (#238). In this release, we've made significant changes to the 'databricks/labs/lsql' directory's 'dashboards.py' file to address formatting breaks in the widget that could occur with Call to Action (CTA) presence in a query. These changes include the addition of new class variables, including _SQL_DIALECT and _DIALECT, and their integration into existing methods such as _parse_header, validate, format, _get_abstract_syntax_tree, and replace_catalog_and_database_in_query. Furthermore, we have developed new methods for creating and deleting schemas and getting the current test purge time. We have also implemented new integration tests to demonstrate the fix for the formatting issue and added new test cases for the query handler's header-splitting functionality, query formatting, and CTE handling. These enhancements improve the library's handling of SQL queries and query tiles in the context of dashboard creation, ensuring proper parsing, formatting, and metadata extraction for a wide range of query scenarios.
• Fixed replace database when catalog or database is None (#237). In this release, we have addressed an issue where system tables disappeared in ucx dashboards when replacing the placeholder database. To rectify this, we have developed a new method, replace_catalog_and_database_in_query, in the dashboards.py file's replace_database function. This method checks if the catalog or database in a query match the ones to be replaced and replaces them with new ones, ensuring that system tables are not lost during the replacement process. Additionally, we have introduced new unit tests in test_dashboards.py to verify that queries are correctly transformed when replacing the database or catalog in the query. These tests include various scenarios, using two parametrized test functions, to ensure the correct functioning of the feature. This change provides a more robust and reliable dashboard display when replacing the placeholder database in the system.

[github-actions[bot]]

❌ 35/36 passed, 1 failed, 2 skipped, 9m57s total

❌ test_dashboards_creates_dashboard_with_replace_database: databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: (1.376s)

``` databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission CREATE on CATALOG. SQLSTATE: 42501 09:24 DEBUG [databricks.sdk] Loaded from environment 09:24 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 09:24 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 09:24 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 09:24 INFO [databricks.sdk] Using Databricks Metadata Service authentication [gw4] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python 09:24 DEBUG [databricks.sdk] Loaded from environment 09:24 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 09:24 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 09:24 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 09:24 INFO [databricks.sdk] Using Databricks Metadata Service authentication 09:24 DEBUG [databricks.sdk] POST /api/2.0/lakeview/dashboards > { > "display_name": "created_by_lsql_ajjWCnhlslgV3QNG" > } < 200 OK < { < "create_time": "2024-07-25T09:24:24.039Z", < "dashboard_id": "01ef4a67ae6f15d7a972ea3aea592dca", < "display_name": "created_by_lsql_ajjWCnhlslgV3QNG", < "etag": "-36911925", < "lifecycle_state": "ACTIVE", < "parent_path": "/Users/4106dc97-a963-48f0-a079-a578238959a6", < "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/created_by_lsql_ajjWCnhlslgV3QNG.lvdash.json", < "serialized_dashboard": "{\"pages\":[{\"name\":\"99642b07\",\"displayName\":\"New Page\"}]}", < "update_time": "2024-07-25T09:24:24.200Z" < } 09:24 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE SCHEMA hive_metastore.lsql_sos7a WITH DBPROPERTIES (RemoveAfter=2024072510) 09:24 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE SCHEMA hive_metastore.lsql_sos7a WITH DBPROPERTIES (RemoveAfter=2024072510) 09:24 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/ > { > "format": "JSON_ARRAY", > "statement": "CREATE SCHEMA hive_metastore.lsql_sos7a WITH DBPROPERTIES (RemoveAfter=2024072510)", > "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID" > } < 200 OK < { < "statement_id": "01ef4a67-aea6-17a4-a0c5-5c1bb113c39d", < "status": { < "error": { < "error_code": "BAD_REQUEST", < "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE on CATA... (20 more bytes)" < }, < "state": "FAILED" < } < } 09:24 DEBUG [databricks.sdk] Loaded from environment 09:24 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred 09:24 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred 09:24 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service 09:24 INFO [databricks.sdk] Using Databricks Metadata Service authentication 09:24 DEBUG [databricks.sdk] POST /api/2.0/lakeview/dashboards > { > "display_name": "created_by_lsql_ajjWCnhlslgV3QNG" > } < 200 OK < { < "create_time": "2024-07-25T09:24:24.039Z", < "dashboard_id": "01ef4a67ae6f15d7a972ea3aea592dca", < "display_name": "created_by_lsql_ajjWCnhlslgV3QNG", < "etag": "-36911925", < "lifecycle_state": "ACTIVE", < "parent_path": "/Users/4106dc97-a963-48f0-a079-a578238959a6", < "path": "/Users/4106dc97-a963-48f0-a079-a578238959a6/created_by_lsql_ajjWCnhlslgV3QNG.lvdash.json", < "serialized_dashboard": "{\"pages\":[{\"name\":\"99642b07\",\"displayName\":\"New Page\"}]}", < "update_time": "2024-07-25T09:24:24.200Z" < } 09:24 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE SCHEMA hive_metastore.lsql_sos7a WITH DBPROPERTIES (RemoveAfter=2024072510) 09:24 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE SCHEMA hive_metastore.lsql_sos7a WITH DBPROPERTIES (RemoveAfter=2024072510) 09:24 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/ > { > "format": "JSON_ARRAY", > "statement": "CREATE SCHEMA hive_metastore.lsql_sos7a WITH DBPROPERTIES (RemoveAfter=2024072510)", > "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID" > } < 200 OK < { < "statement_id": "01ef4a67-aea6-17a4-a0c5-5c1bb113c39d", < "status": { < "error": { < "error_code": "BAD_REQUEST", < "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE on CATA... (20 more bytes)" < }, < "state": "FAILED" < } < } 09:24 DEBUG [databricks.sdk] DELETE /api/2.0/lakeview/dashboards/01ef4a67ae6f15d7a972ea3aea592dca < 200 OK < {} [gw4] linux -- Python 3.10.14 /home/runner/work/lsql/lsql/.venv/bin/python ```

_{Running from acceptance #340}