mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

Passwords included in log output #65

Closed harrybiddle closed 2 years ago

harrybiddle commented 2 years ago

When using the ExecuteSQL command with a PostgreSQL database, passwords are leaked to the logs:

image

It would be better to pass sensitive values using Popen's env option.

~(If you also passed in the query as the process' stdin, you wouldn't have to worry about quoting with shlex.quote):~

jankatins commented 2 years ago

This doesn't quite work as we execute a whole pipe and so the env variable can be on both sides (and with different passwords).

There is mara_pipelines.config.password_masks() which will replace everything returned by this function before it is shown/persisted: https://github.com/mara/mara-pipelines/blob/master/mara_pipelines/logging/logger.py#L29-L32

harrybiddle commented 2 years ago

Great, thanks! This is working for me.