[Bug] Compiled files in run artifacts are showing incorrect schema

[crystalro0]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

While doing a blue/green test, user is adding a 'staging_' prefix with the generate_schema_name macro. The compiled files within the run artifacts.

• vars:
  • is_stg (value of 1 passed through dbt run --vars 'is_stg: 1')
  • bg_schemas (in dbt_project.yml)

{%- if custom_schema_name in var('bg_schemas') and var('is_stg', none) is not none -%}
    {%- set custom_schema_name = 'staging_' ~ custom_schema_name -%}
    {{ custom_schema_name | trim }}
{%- else -%}
    {{ custom_schema_name | trim }}
{%- endif -%}

Expected Behavior

'staging_' prefix should be displayed for those schemas identified in the blue/green schema var.

Steps To Reproduce

1. Create generate_schema_name macro with blue/green schemas and is_stg var.
2. Identify blue/green schemas var in dbt_project.yml.
3. Create job with the following run steps: dbt run --vars 'is_stg: 1 && dbt test --vars 'is_stg: 1
4. Run job and check the compiled file for a blue/green schema.
5. Should display 'staging_' prefix

Relevant log output

No response

Environment

- dbt: 0.21.0

What database are you using dbt with?

redshift

Additional Context

No response

[emmyoop]

@crystalro0 sorry for the delay in getting back to you. Thank you for the detailed write up on what a user is experiencing.

Unfortunately, I'm having trouble replicating the experience or maybe I'm misinterpreting the problem. I set up a (very) simple project with just 2 models, the bg_schemas var, and your custom generate_schema_name macro. When I run it the appropriate schemas get the 'staging_' prefix. I'm running in dbt Cloud with 0.21.0 against Redshift with a job as you indicated.

The appropriate models end up in a schema prefixed with 'staging' and the others do not. And when I look into target/compiled/ at the model files, the refs have been converted into the expected schema prefixed with 'staging'.

Would you be able to take a look at the project I set up and let me know if there is something that I missed setting up? If not, would you be able to fork it and run it yourself to see if you get the same unexpected results?

This is going to take a bit more digging to get to the bottom of but we'll figure it out!

One more note that there were multiple fixes that came out in 0.21.1 that should not be breaking and might be worth the upgrade.

[crystalro0]

hi @emmyoop, thanks for checking this out! Sorry I didn't specify which compiled files. My bad on that. The original schema is showing up in the compiled schema tests. Under my local: compiled/jaffle_shop/models/staging/jaffle_shop/stg_jaffle_shop.yml/schema_test/accepted_values_stg_payments.sql

Edit: but I'm seeing the staging_ prefixed in your schema_test compiled folder...so arg. Not seeing anything really different setup-wise.

[gshank]

This looks like what will happen with dbt 0.21 and partial parsing turned on and a custom generate_schema_name macro. It is fixed in 1.0.0 (where the fix consists of forcing a full re-parse). If you add or change a custom generate_schema_name macro in 0.21 you will have to do a full-reparse by turning off partial parsing.

[emmyoop]

@crystalro0 since @gshank pointed out this is fixed in 1.0.0, I'm going to go ahead and close this issue as we will not be fixing it in 0.21. Please reopen it if you find that it is still a bug in 1.0!