MattTriano / analytics_data_where_house

An analytics engineering sandbox focusing on real estates prices in Cook County, IL
https://docs.analytics-data-where-house.dev/
GNU Affero General Public License v3.0
9 stars 0 forks source link

Sort out issues with permissions errors involving directories and files created by Airflow #26

Closed MattTriano closed 1 year ago

MattTriano commented 1 year ago

This kind of error can be reproduced by, for example, running this version of the clean_dbt makefile recipe that will delete the /airflow/dbt/target/ directory (which contains compiled dbt models/resources). Then run a DAG that includes a dbt task, which will recreate the /airflow/dbt/target/ and then try to write compiled models to that directory, but the permissions of the newly created directory won't allow that write.

It's simple enough to just tack on mkdir airflow/dbt/target to the clean_dbt recipe, but I want to use this issue to track appearances of this issue, help reliably understand the cause, and account for it.