As part of the work to get a more flexible approach, and a more consistent approach across warehouses, we would like to allow users to add their own custom contexts to the events this run table for redshift and postgres. We first need to move all our contexts to this table, and currently the new de-dupe logic makes thisa quite a lot of effort and will involve writing many (optional) subqueries.
The macro added in this PR allows for a more consistent and scalable approach in the future, with all columns from a context/sde pulled through in the query, excluding the schema information columns (due to expected name clashes) and the id and timestamp columns which are aliased with the schema name to avoid column name clashes.
The macro will just be available in this verison and later versions of web/mobile can make use of it.
There is an open question of would you ever want the schema columns included, if so they can be prefixed in the same way, I'm not too sure if people would need this? These column names are a little longer is my only worry that we might hit the column name limit for postgres/redshift (59 and 115 respectively I think).
Tests and docs added for the macro.
Checklist
[x] I have verified that these changes work locally
[x] I have updated the README.md (if applicable)
[x] I have added tests & descriptions to my models (and macros if applicable)
Description & motivation
As part of the work to get a more flexible approach, and a more consistent approach across warehouses, we would like to allow users to add their own custom contexts to the events this run table for redshift and postgres. We first need to move all our contexts to this table, and currently the new de-dupe logic makes thisa quite a lot of effort and will involve writing many (optional) subqueries.
The macro added in this PR allows for a more consistent and scalable approach in the future, with all columns from a context/sde pulled through in the query, excluding the schema information columns (due to expected name clashes) and the id and timestamp columns which are aliased with the schema name to avoid column name clashes.
The macro will just be available in this verison and later versions of web/mobile can make use of it.
There is an open question of would you ever want the schema columns included, if so they can be prefixed in the same way, I'm not too sure if people would need this? These column names are a little longer is my only worry that we might hit the column name limit for postgres/redshift (59 and 115 respectively I think).
Tests and docs added for the macro.
Checklist