Use as a pre-hook rather than run-operation on Databricks

ghost commented 2 years ago

Describe the bug

When the macro is run from the on-run-start hook (like described here : https://github.com/dbt-labs/dbt-external-tables/issues/34) , it causes in an error on dbt-databricks. When I run 'dbt run-operation stage_external_sources' there are no errors.

In line 22 of plugins\spark\get_external_build_plan.sql a module reference is missing.

Steps to reproduce

install dbt-code and dbt-databricks. Add and external source to the project. Add this to project yml

on-run-start:

"{{ dbt_external_tables.stage_external_sources() }}"

Expected results

The stage_external_sources would run during 'dbt run' without any problems.

Actual results

The following error is thrown:

Database error while running on-run-start Encountered an error: Compilation Error in operation fnp-on-run-start-0 (.\dbt_project.yml) 'spark__recover_partitions' is undefined

in macro get_external_build_plan (macros\common\get_external_build_plan.sql) called by macro stage_external_sources (macros\common\stage_external_sources.sql) called by macro spark__get_external_build_plan (macros\plugins\spark\get_external_build_plan.sql) called by operation fnp-on-run-start-0 (.\dbt_project.yml)

Screenshots and log output

n.a.

System information

packages:

package: dbt-labs/dbt_external_tables version: 0.8.0
package: dbt-labs/dbt_utils version: 0.8.6
package: dbt-labs/spark_utils
version: 0.3.0

Which database are you using dbt with?

databricks and delta tables on Azure storage account.

The output of dbt --version:

Core:
  - installed: 1.1.0
  - latest:    1.1.1 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.1.0 - Up to date!
  - spark:      1.1.0 - Up to date!

The operating system you're using: Windows 11

The output of python --version: Python 3.9.4

Additional context

n.a.

github-actions[bot] commented 1 year ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 1 year ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

dbt-labs / dbt-external-tables