chronitis
commented
2 years ago Worth noting that if using a static external location (ie, s3_data_naming=schema_table), this is affected by #54, so after #43 non-incremental tables don't get cleaned up, so seeds get their data duplicated and non-incremental tables will fail with a warning that there is data in the prefix already.
Either reverting #43 or merging #49 should fix this. s3_data_naming=uuid isn't affected since it won't try and reuse old prefixes.
jessedobbelaere
mrshu
VDFaller
rumbin
mattiamatrix
nicor88
Problem
Currently, the only options for determining where data ends up in S3 are to set
s3_staging_bucketin the connection properties or setexternal_locationon each model.The
s3_staging_bucketargument is ignored if the Athena workgroup already has a staging bucket configured (but is used bydbt seed, which always sets a location).We have a (not uncommon?) layout of one S3 bucket for the results of all Athena queries from which objects are rapidly expired, and another with appropriate lifecycle configuration for storing created tables we want to keep.
Aside from setting
external_locationon every model (which isn't so easily composable across different profiles), there isn't a nice way to control S3 layout indbt-athenaas it stands.Implementation
This adds two new (optional) connection options:
s3_data_dir: if set, the default root directory to create tables and seeds in (eg,s3://my-data-bucket/dbt/)s3_data_naming: the strategy for naming subdirectories ofs3_data_dirin which we'll actually store tables; two implemented options:uuid: tables or seeds are saved to{s3_data_dir}/{uuid4}/(this was how seed tables were already named)schema_table: name according to the schema (ie, Athena database) and table like{s3_data_dir}/{schema}/{table}/If
s3_data_diris unset, the behaviour should be unchanged:{s3_staging_dir}/tables/{uuid4}/external_locationif set, and Athena's default otherwise.