readme for runing dataset_ingress_v1, plus SKIP_DATA_UPLOAD capability - Githubissues

ONSdigital / dp-data-pipelines

Pipeline specific python scripts and tooling for automated website data ingress.

MIT License

1 stars 0 forks source link

readme for runing dataset_ingress_v1, plus SKIP_DATA_UPLOAD capability #140

Closed mikeAdamss closed 2 months ago

mikeAdamss commented 2 months ago

What

Added instructions for running dataset_ingress_v1.py.
Added a new env var of SKIP_DATA_UPLOAD as we probaly dont want to upload files to s3 every time we run a transform we're developing.
Fixed a mistake of mine where I said we dont need to raise errors to log them, we do, the logger needs the stack trace.

How to review

Sanity check - run the script as per instructions and make sure it works for you.

Who can review

Anyone.