Snowflake-Labs / schemachange

A Database Change Management tool for Snowflake
Apache License 2.0
502 stars 225 forks source link

Jinja Not Rendered in schemachange-config.yml #134

Closed ATidmore closed 1 year ago

ATidmore commented 1 year ago

I need to parameterize my change-history-table based on the target environment and cannot successfully get this Jinja to resolve when ever I deploy.

change-history-table: {% if TARGET_ENV != "PROD" %}{{TARGET_ENV}}_{% endif %}EDA_ENGINEERING.SCHEMACHANGE.CHANGE_HISTORY

This works successfully in my change scripts, and I see that in cli.py the config file should be getting passed to the Jinja engine. Not sure why this works elsewhere, just not in my config file.

MACKAT05 commented 1 year ago

does this correction work? change-history-table: {% if TARGET_ENV != "PROD" %}{{ TARGET_ENV }}_{% endif %}EDA_ENGINEERING.SCHEMACHANGE.CHANGE_HISTORY
The mandatory spaces around the variable reference and the curly braces were missing from your posted comment. This could just be a transcription error.

ATidmore commented 1 year ago

No, unfortunately the jinja exception is still raised that TARGET_ENV is undefined. Removing that jinja from my config file resolves the exception, but not my need to parametrize the environment :)

MACKAT05 commented 1 year ago

I believe we have a 🐔🥚. VARS that would contain the value for TARGET_ENV is loaded after the yaml is passed to the parser. could you perhaps use something like this line: https://github.com/Snowflake-Labs/schemachange/blob/e8fee3dfb36af38e689a71764096fbe1b9c73fa2/demo/citibike_jinja/schemachange-config.yml#L6 as an alternate route to introduce that information into the workflow? change-history-table: {{env_var('TARGET_ENV', '')}}EDA_ENGINEERING.SCHEMACHANGE.CHANGE_HISTORY Then Set the value of TARGETENV as an environment variable with the value : `TEST`DEV_ STAGE_ as needed?

The TARGET_ENV var value for schemachange can be populated in the Yaml the same way so that the downstream rendering of the .sql files can be retained without altering them all to point to the environment variable.

sfc-gh-tmathew commented 1 year ago

Hello @ATidmore

Did @MACKAT05's suggestion address your need to parameterize the yaml file ?

@MACKAT05, Could you raise the PR corresponding to this issue?

MACKAT05 commented 1 year ago

For reference this is my clunky solution implementation to much the same problem...
image the Yaml order doesn't matter so we could probably include a more explicit example of handling this in an example yaml with the "snowflake-role", "snowflake-account","snowflake-database", "change-history-table" clustered together wrapped in an if statement.

ATidmore commented 1 year ago

Thanks @MACKAT05 - I was coming along in my understanding of Jinja and env vars (and DBT and Schemachange etc. etc.) over the past few months, so coupling an environment variable with some jinja should do the trick.