This PR refactors how we are inferring the database name for create-a-derived-table databases created in athena.
Previously we took the filename and replicated the logic used in cadet prod to parse these into {database_name}.{table_name}
However if looking at the dev env of cadet, different logic is applied when parsing the filenames to get database names, with _dev_dbt suffix added to every database name. The post parse db name is available for each node under the schema key and this PR switches our custom source and transformers to create the database name according to the value in the schema key.
This PR refactors how we are inferring the database name for create-a-derived-table databases created in athena.
Previously we took the filename and replicated the logic used in cadet prod to parse these into
{database_name}.{table_name}However if looking at the dev env of cadet, different logic is applied when parsing the filenames to get database names, with
_dev_dbtsuffix added to every database name. The post parse db name is available for each node under theschemakey and this PR switches our custom source and transformers to create the database name according to the value in theschemakey.