Read Only File System with MWAA 2.5.1

[kundankumar2021]

We are using airflow-dbt-python to orchestrate dbt commands and running Airflow in mwaa.

Version used: Amazon MWAA=2.5.1 airflow-dbt-python=0.13.2 dbt-core=1.3.4

However this is throwing below error.

[2023-07-27, 17:34:28 UTC] {{taskinstance.py:1768}} ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow_dbt_python/operators/dbt.py", line 166, in execute with self.dbt_directory() as dbt_dir: # type: str File "/usr/lib/python3.10/contextlib.py", line 142, in exit next(self.gen) File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow_dbt_python/operators/dbt.py", line 284, in dbt_directory self.dbt_hook.push_dbt_project( File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow_dbt_python/hooks/dbt.py", line 540, in push_dbt_project return backend.push_dbt_project( File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow_dbt_python/hooks/backends/base.py", line 89, in push_dbt_project self.push_many( File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow_dbt_python/hooks/backends/localfs.py", line 117, in push_many shutil.copytree( # type: ignore File "/usr/lib/python3.10/shutil.py", line 558, in copytree return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, File "/usr/lib/python3.10/shutil.py", line 512, in _copytree raise Error(errors) shutil.Error: [('/tmp/airflowtmps1gh6wxo/model/staging', '/usr/local/airflow/dags/test_dag/model/staging', '[Errno 30] Read-only file system'), ('/tmp/airflowtmps1gh6wxo/model', '/usr/local/airflow/dags/test_dag/model', '[Errno 30] Read-only file system'), ('/tmp/airflowtmps1gh6wxo/dbt_packages', '/usr/local/airflow/dags/test_dag/dbt_packages', "[Errno 30] Read-only file system: '/usr/local/airflow/dags/test_dag/dbt_packages'"), ('/tmp/airflowtmps1gh6wxo/target/compiled/test_dag/model/staging', '/usr/local/airflow/dags/test_dag/target/compiled/test_dag/model/staging', '[Errno 30] Read-only file system'), ('/tmp/airflowtmps1gh6wxo/target/compiled/test_dag/model', '/usr/local/airflow/dags/test_dag/target/compiled/test_dag/model', '[Errno 30] Read-only file system'), ('/tmp/airflowtmps1gh6wxo/target/compiled/test_dag', '/usr/local/airflow/dags/test_dag/target/compiled/test_dag', '[Errno 30] Read-only file system'), ('/tmp/airflowtmps1gh6wxo/target/compiled', '/usr/local/airflow/dags/test_dag/target/compiled', '[Errno 30] Read-only file system'), ('/tmp/airflowtmps1gh6wxo/target', '/usr/local/airflow/dags/test_dag/target', '[Errno 30] Read-only file system'), ('/tmp/airflowtmps1gh6wxo', '/usr/local/airflow/dags/test_dag', '[Errno 30] Read-only file system')] [2023-07-27, 17:34:28 UTC] {{taskinstance.py:1318}} INFO - Marking task as FAILED. dag_id=test_dag, task_id=dbt_compile, execution_date=20230727T173419, start_date=20230727T173426, end_date=20230727T173428

[drackham]

I ran into a similar error recently. For our configuration I was able to add upload_dbt_project=False to the operator's args and that resolved it.