dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
313 stars 123 forks source link

stage_external_sources Comparing source_name of the node instead of the name of the node #309

Closed ward-resa closed 3 weeks ago

ward-resa commented 3 months ago

Describe the bug

When calling the stage_external_sources macro and passing the source name as a list of values, when the name does not contain a '.' it is comparing the node.source_name instead of the node.name for the selected source.

Steps to reproduce

In my pre-hook I am can call the macro with the select parameter with the schema.source_name and it works. If I just call it with the source_name it does not.

Expected results

Calling the macro with the schema or without the schema should still find the source by name.

Actual results

Ignores the source if only the name is used in the select parameter.

Which database are you using dbt with?

The output of dbt --version:

Core:
  - installed: 1.8.4
  - latest:    1.8.4 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

The operating system you're using:

macOS 14.5

The output of python --version:

Python 3.11.9

Additional context

I believe that Line 20 in the macro stage_external_sources is the culprit. I think that the following would fix the issue. Unless I am misinterpreting the intention of that line of code.

existing code

                        {% if src == node.source_name %}

proposed change

                        {% if src == node.name %}
laurianneol commented 4 days ago

Hi,

I've encountered an issue with the stage_external_sources macro in the latest release. The problem arises from this change. Because of this change, I'm unable to run the command dbt run-operation stage_external_sources --args '{"select": "source_name"}'. With this command line, I was previously able to drop and create all the tables from my source (using ext_full_refresh = true).

Could you advise on how I can run all tables using the current setup?

Locally, I added an argument to the macro to use tags that I had assigned to my sources, and it works perfectly. Could you please look into this issue and consider incorporating a similar solution?

I'm new to GitHub, so any guidance or help would be greatly appreciated!

Thanks!