After switching to using information_schema where possible to get column types to address the issue of describe truncation them, I discovered that information_schema is off out of sync when this call is made. I discovered this by running an incremental job twice. On the second run, the columns for some tables from the first run were available in information_schema. This PR forces metadata sync before getting column information to ensure the information_schema is up to date. We should probably use this technique more pervasively, but I do want to hold off on that until we have behavior flags, since it could have a performance impact.
Also, commenting out behavior flags for now, because support doesn't exist yet in dbt-core 1.8
Checklist
[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.
Description
After switching to using information_schema where possible to get column types to address the issue of describe truncation them, I discovered that information_schema is off out of sync when this call is made. I discovered this by running an incremental job twice. On the second run, the columns for some tables from the first run were available in information_schema. This PR forces metadata sync before getting column information to ensure the information_schema is up to date. We should probably use this technique more pervasively, but I do want to hold off on that until we have behavior flags, since it could have a performance impact.
Also, commenting out behavior flags for now, because support doesn't exist yet in dbt-core 1.8
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.