search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.49k stars 14.13k forks source link

Redact secrets from external secrets provider #42420

Open stuart23 opened 4 days ago

stuart23 commented 4 days ago

Apache Airflow version

2.10.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When using a third party secrets backend (e.g. AWS Secrets Manager), secret values are not redacted from task logs and appear as plaintext in the logs.

What you think should happen instead?

When writing task logs, the worker should list all the secret values from the backend and add that to the list of things to redact.

How to reproduce

Connect a secrets backend e.g. AWS Secrets Manager Print a value from the backend - the plaintext will appear in the task logs

Operating System

N/A

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

wseaton commented 2 days ago

Not just secret values, it is also important that bootstrap variables for secrets manager configuration like AIRFLOW__SECRETS__BACKEND_KWARGS can be masked in task logs, since they are visible to the workers.