This PR will address the following Issue/Feature: Issue #120
This PR will result in the following new package version:v1.7.2
This will be non breaking as it will only adjust the regex search in the Datarbricks SQL Warehouse to remove a leading backslash. There will be no breaking changes resulting from this release.
Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:
Bug Fixes
Removal of the leading / from the target.http_path regex search within the is_databricks_sql_warehouse() macro to accurately identify SQL Warehouse Databricks destinations in Quickstart.
The macro above initially worked as expected in dbt core environments; however, in Quickstart implementations this data model was not working. This was due to Quickstart removing the leading / from the target.http_path. Thus resulting in the regex search to always fail.
PR Checklist
Basic Validation
Please acknowledge that you have successfully performed the following commands locally:
[X] dbt run –full-refresh && dbt test
[X] dbt run (if incremental models are present) && dbt test
Before marking this PR as "ready for review" the following have been applied:
[X] The appropriate issue has been linked, tagged, and properly assigned
[X] All necessary documentation and version upgrades have been applied
[x] docs were regenerated (unless this PR does not include any code or yml updates)
[x] BuildKite integration tests are passing
[x] Detailed validation steps have been provided below
Detailed Validation
Please share any and all of your validation steps:
In order to test this I did the following:
Understood why the v1.7.1 release is not working as expected. It is due to code on the Quickstart side which removes the leading / from the target.http_path. Please see internal messages for link to code. Therefore, removing the leading / from the regex search in the is_databricks_sql_warehouse() macro should do the trick with working on both dbt core and Quickstart.
I received confirmation from engineering that this should work as expected.
After applying the update to the macro I tested to ensure the logic still works as expected on a Databricks SQL Warehouse destination. You can see in the below screenshot that after running dbt run -t databricks-sql (after a full refresh to simulate if there was any incremental strategy) there was no incremental logic included in the model. Further, no incremental models were captured in the end complete statement.
To ensure the logic is still working as expected for the Databricks All Purpose Cluster, I performed the same steps to ensure the incremental strategy was being applied. I was able to confirm this is the case!
Lastly, I wanted to ensure that there is no change for a non Databricks destination. While no changes were made that should impact any other destination, I did still want to check at least two. I checked Snowflake and BigQuery. Both looked to be in order and utilizing the incremental strategy on subsequent loads!
BigQuery
Snowflake
If you had to summarize this PR in an emoji, which would it be?
PR Overview
This PR will address the following Issue/Feature: Issue #120
This PR will result in the following new package version:
v1.7.2This will be non breaking as it will only adjust the regex search in the Datarbricks SQL Warehouse to remove a leading backslash. There will be no breaking changes resulting from this release.
Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:
PR Checklist
Basic Validation
Please acknowledge that you have successfully performed the following commands locally:
Before marking this PR as "ready for review" the following have been applied:
Detailed Validation
Please share any and all of your validation steps:
In order to test this I did the following:
v1.7.1release is not working as expected. It is due to code on the Quickstart side which removes the leading/from thetarget.http_path. Please see internal messages for link to code. Therefore, removing the leading/from the regex search in theis_databricks_sql_warehouse()macro should do the trick with working on both dbt core and Quickstart.dbt run -t databricks-sql(after a full refresh to simulate if there was any incremental strategy) there was no incremental logic included in the model. Further, no incremental models were captured in the end complete statement.If you had to summarize this PR in an emoji, which would it be?
◀️