This PR simplifies, restructures, and cleans up our data architecture repository. Don't be fooled by the absolutely heinous diff, this is mostly just moving things around. I recommend reviewing this PR by just switching branches and browsing the new structure. The main changes are:
Remove the aws-athena directory and all symlinks to that directory (replacing with hard copies instead). This simplifies dbt model setup considerably. I've also updated the documentation to reflect the change.
Add a top-level README to signpost to other documentation.
Remove all RPIE-related models and assets. The data will still exist in S3 if it's needed.
Update various miscellaneous files like .gitignores and pre-commit configs
Remove other deprecated files, such as .gitkeep files and our old data catalog
Rename the aws-s3 directory to just etl
I'll call out other changes with PR comments. Note that merging this may rebuild the entire dbt DAG and will likely create substantial merge conflicts.
wrridgeway
commented
1 month ago
This PR simplifies, restructures, and cleans up our data architecture repository. Don't be fooled by the absolutely heinous diff, this is mostly just moving things around. I recommend reviewing this PR by just switching branches and browsing the new structure. The main changes are:
aws-athenadirectory and all symlinks to that directory (replacing with hard copies instead). This simplifies dbt model setup considerably. I've also updated the documentation to reflect the change.aws-s3directory to justetlI'll call out other changes with PR comments. Note that merging this may rebuild the entire dbt DAG and will likely create substantial merge conflicts.
Also closes #148.