Open tade0726 opened 7 months ago
This is expected behavior, as python models are integrated into the rest of your dbt project using SQL (for example, on an incremental model, the merge behavior is conducted in SQL), and that SQL would be executed on the AP Cluster. We are investigating ways for python model behavior to be more 'spark-like', but for now I would say this is an enhancement request, rather than a bug, as it is consistent with the structure imposed by dbt-core.
Thanks, Benc. It clears my doubts.
@benc-db Would it be possible to use a more simple approach when running a python model inside a job cluster like following:
dbutils.notebook.run("....")
(see Run a Databricks notebook from another notebook) inside a own processI am not sure but it looks to me, that the strict seperation between execution (dbt python code) and the model execution (putting model into an isolated space) seems to be a bit oversized on Databricks job clusters, because the job will run nevertheless on spark on the master node. But maybe I am not getting the full picture of this issue...
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.
Describe the bug
When I try the feature of running jobs on the job cluster instead of the all-purpose cluster, both the purpose cluster and the job cluster trigger, and only when the all-purpose starts, and the the job cluster follows.
Steps To Reproduce
profiles.yaml
dbt_project.yml
, follow instructions (https://docs.getdbt.com/docs/build/python-models#specific-data-platforms)Expected behavior
Once configured job cluster detail provided, all purpose cluster should not be trigged.
Screenshots and log output
System information
The output of
dbt --version
:The operating system you're using:
Mac OS, should be irrelevant
The output of
python --version
:Python 3.10.13
Additional context
NaN