search

transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

Do not log redshift credentials to stdout #9

Closed Limess closed 4 years ago

Limess commented 5 years ago

Issue

Currently credentials are logged to stdout:

COPY public.my_table ("_SDC_BATCHED_AT", "_SDC_DELETED_AT", "_SDC_EXTRACTED_AT", "SOME_COLUMN", "PK", "UPDATED_AT") FROM 's3://my_s3_bucket/__tmp/pipelinewise_public-my_table_20190918-105311-945010.csv'
ACCESS_KEY_ID '*********************'
SECRET_ACCESS_KEY '****************************************'
DELIMITER ',' REMOVEQUOTES ESCAPE
BLANKSASNULL TIMEFORMAT 'auto'
COMPUPDATE OFF STATUPDATE OFF

Where the * characters are actual credentials. This is undesirable as it means the keys are propagated to other logging systems, e.g airflow in our case.

Expected

Credentials are redacted, likely as * characters.

koszti commented 5 years ago

Fixed by https://github.com/transferwise/pipelinewise-target-redshift/commit/01e9ea146b6885c31e67c6f144b38e30e5e43ed7.

Generated SQLs are now only visible in debug mode and not in the default info level.

If you use target-redshift alone: You'll not see credentials in the log any more If you use target-redshift with PipelineWise: You'll still see redshift credentials in some logs when FastSync to Redshift is triggered. Enhancing logging is currently in progress and keep this issue open until it's not fixed everywhere in PipelineWise.