Gdxdsd 5797 set up data feed for tran data

[doughon]

This PR does the following: Adds 5 sets of files for monthly automated webdata for TRAN. This includes converting 4 existing manual feeds to the automated process and the creation of a new 5th feed.

Please note that some of the config file were modified so that the tests can run. The modified configs will be added to the ticket in a zip file.

Testing instructions:

1. Review the DML files and compare with the original SQL to verify that no changes were introduced (only changes should be wrapping of query in SELECT *, escaped single quotes in the file and formatting)
   • Internet assets
   • • webdata_tran_assetsinternet_monthly.sql
   • • https://analytics.gov.bc.ca/sql/wcpfsftvpfdx6d (same as SQL as this is the first time creating of this data feed, not previous work)
   • Intranet assets
   • • webdata_tran_assetstrannet_monthly.sql
   • • https://analytics.gov.bc.ca/sql/ddgb4rrms2kwd5
   • GovTran
   • • webdata_tran_govtran_monthly.sql
   • • https://analytics.gov.bc.ca/sql/krtpskdfwxm7rp
   • TranBC
   • • webdata_tran_tranbc_monthly.sql
   • • https://analytics.gov.bc.ca/sql/hgqz8s4tcfp8qc
   • TranNet
   • • webdata_tran_trannet_monthly.sql
   • • https://analytics.gov.bc.ca/sql/bf6ymwdq2nmk9d
2. Review the config files
   • webdata_tran_assetsinternet_monthly.json
   • webdata_tran_assetstrannet_monthly.json
   • webdata_tran_govtran_monthly.json
   • webdata_tran_tranbc_monthly.json
   • webdata_tran_trannet_monthly.json
3. Log into the ec2 instance through the following commands

   awsmfa prod <AWS OTP>
   microservice_ssm
   cd /home/microservice/branch/GDXDSD-5797-set-up-data-feed-for-tran-data/redshift_to_s3

4. Run the following command and compare its output to what's expected.

   pipenv run python redshift_to_s3.py -c config.d/webdata_tran_assetsinternet_monthly-test.json
   pipenv run python redshift_to_s3.py -c config.d/webdata_tran_assetstrannet_monthly-test.json
   pipenv run python redshift_to_s3.py -c config.d/webdata_tran_govtran_monthly-test.json
   pipenv run python redshift_to_s3.py -c config.d/webdata_tran_tranbc_monthly-test.json
   pipenv run python redshift_to_s3.py -c config.d/webdata_tran_trannet_monthly-test.json

   
   ***The microservice ran successfully***

Report: redshift_to_s3.py

Config: config.d/webdata_tran_assetsinternet_monthly-test.json

DML: webdata_tran_assetsinternet_monthly.sql

Microservice started at: 2023-08-31 11:09:13-0700 (PDT), ended at: 2023-08-31 11:09:15-0700 (PDT), elapsing: 0:00:02.216543.

Objects loaded to S3 /batch: 1/1 Objects successfully loaded to S3 /batch: 1

List of objects successfully loaded to S3 /batch

1. processed/batch/client/doug_test/GDXDSD-5797/webdata_tran/assetsinternet_monthly/v02-Aug_2023_automated/webdata_tran_assetsinternet_monthly_20230831T180913

Objects to store: 1 Objects stored to s3 /client: 1

List of objects stored to S3 /client: 1: client/doug_test/GDXDSD-5797/webdata_tran/assetsinternet_monthly/v02-Aug_2023_automated/webdata_tran_assetsinternet_monthly_20230831T180913_part000.csv

Objects to process: 1 Objects processed to s3 /good: 1

List of objects processed to S3 /good: 1: processed/good/client/doug_test/GDXDSD-5797/webdata_tran/assetsinternet_monthly/v02-Aug_2023_automated/webdata_tran_assetsinternet_monthly_20230831T180913_part000

5. Check to see if the file appear in the s3 processed batch bucket: 
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/batch/client/doug_test/GDXDSD-5797/webdata_tran/assetsinternet_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/batch/client/doug_test/GDXDSD-5797/webdata_tran/assetstrannet_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/batch/client/doug_test/GDXDSD-5797/webdata_tran/govtran_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/batch/client/doug_test/GDXDSD-5797/webdata_tran/tranbc_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/batch/client/doug_test/GDXDSD-5797/webdata_tran/trannet_monthly/v02-Aug_2023_automated/&showversions=false
6. Check to see if the file appear in the s3 client bucket: 
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=client/doug_test/GDXDSD-5797/webdata_tran/assetsinternet_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=client/doug_test/GDXDSD-5797/webdata_tran/assetstrannet_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=client/doug_test/GDXDSD-5797/webdata_tran/govtran_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=client/doug_test/GDXDSD-5797/webdata_tran/tranbc_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=client/doug_test/GDXDSD-5797/webdata_tran/trannet_monthly/v02-Aug_2023_automated/&showversions=false
7. Check to see if the file appear in the s3 processed good bucket: 
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/good/client/doug_test/GDXDSD-5797/webdata_tran/assetsinternet_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/good/client/doug_test/GDXDSD-5797/webdata_tran/assetstrannet_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/good/client/doug_test/GDXDSD-5797/webdata_tran/govtran_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/good/client/doug_test/GDXDSD-5797/webdata_tran/tranbc_monthly/v02-Aug_2023_automated/&showversions=false
- https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&prefix=processed/good/client/doug_test/GDXDSD-5797/webdata_tran/trannet_monthly/v02-Aug_2023_automated/&showversions=false

[bradleyoldham]

Reviewed and working according to plan. Please merge and clean-up as required.