Dbt fails to compile when trying to create external table with models that are referencing it.

juls858 commented 2 years ago

Describe the bug

creating a model with a reference to the source causing dependency failure when using run-operation

Steps to reproduce

create model referencing source:

select *
FROM
    {{ source(
        "ethereum_external_bronze",
        "token_balances"
    ) }}

sources:
  - name: ethereum_external_bronze
    schema: bronze
    loader: S3
    tables:
      - name: token_balances

❯ dbt run-operation --target dev stage_external_sources --vars "ext_full_refresh: true" --args "select: ethereum_external_bronze.token_balances"

Expected results

Expected external table to be created.

Actual results

❯ dbt run-operation --target dev stage_external_sources --vars "ext_full_refresh: true" --args "select: ethereum_external_bronze.token_balances"                                                                                 
17:53:35  Running with dbt=1.0.4
17:53:35  Partial parse save file not found. Starting full parse.
17:53:42  Encountered an error:
Compilation Error in operation ethereum_models-on-run-start-2 (./dbt_project.yml)
  Operation 'operation.ethereum_models.ethereum_models-on-run-start-2' (./dbt_project.yml) depends on a node named 'streamline__token_balances_realtime' which was not found

Screenshots and log output

System information

The contents of your packages.yml file:

Which database are you using dbt with?

[ ] redshift
[ ] snowflake
[ ] other (specify: ____)

The output of dbt --version:

<output goes here>

The operating system you're using:

The output of python --version:

Additional context

jeremyyeo commented 2 years ago

@juls858 there appear to be no issue with this toy example:

My guess is you have got something else going on with your project given the error is happening in an on-run-start hook. Can you provide what your on-run-start hooks are?

Additionally, if I ref a model that doesn't actually exist exist in my project in my on-run-start:

So you probably have an on-run-start op that's reffing streamline__token_balances_realtime but that streamline__token_balances_realtime doesn't actually exist in your project - that is to say, this is not an issue of the dbt-external-tables package is my bet.

Jay-code0 commented 1 year ago

A missing file might be the cause of the issue, the below request if created would allow more eligant handling of errors such as this, https://github.com/dbt-labs/dbt-external-tables/issues/174

github-actions[bot] commented 1 year ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 1 year ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

dbt-labs / dbt-external-tables