bcgov / GDX-Analytics-microservice

The collection of GDX-Analytics Python microservices used to load and process data between systems and services.
Apache License 2.0
2 stars 0 forks source link

Updated dmls to use the new stable name PDT #215

Closed doughon closed 6 months ago

doughon commented 6 months ago

This PR does the following:

Updates the following DMLs to use the newly updated stable name PDTs :

Please note that some of the config file were modified so that the tests can run. The modified configs will be added to the ticket in a zip file.

Testing instructions:

  1. Log into the ec2 instance through the following commands
    awsmfa prod <AWS OTP>
    microservice_ssm
    cd /home/microservice/branch/GDXDSD-6969_update_redshift_to_s3_stable_name_pdts/redshift_to_s3
  2. Run the following command and compare its output to what's expected.
    pipenv run python redshift_to_s3.py -c config.d/webdata_healthgateway_app_ratings_daily.json
    pipenv run python redshift_to_s3.py -c config.d/webdata_healthgateway_app_ratings_monthly.json 
    pipenv run python redshift_to_s3.py -c config.d/webdata_healthgateway_downloaded_cards_daily.json 
    pipenv run python redshift_to_s3.py -c config.d/webdata_healthgateway_downloaded_cards_monthly.json 
    pipenv run python redshift_to_s3.py -c config.d/webdata_healthgateway_downloaded_reports_daily.json 
    pipenv run python redshift_to_s3.py -c config.d/webdata_healthgateway_downloaded_reports_monthly.json 
    
    ***The microservice ran successfully***

Report: redshift_to_s3.py

Config: config.d/webdata_healthgateway_downloaded_reports_monthly.json

DML: webdata_healthgateway_downloaded_reports_monthly.sql

Microservice started at: 2024-05-07 10:55:19-0700 (PDT), ended at: 2024-05-07 10:55:20-0700 (PDT), elapsing: 0:00:01.073265.

Objects loaded to S3 /batch: 1/1 Objects successfully loaded to S3 /batch: 1

List of objects successfully loaded to S3 /batch

  1. processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/webdata_healthgateway_downloaded_reports_monthly_20240507T175519

Objects to store: 1 Objects stored to s3 /client: 1

List of objects stored to S3 /client: 1: client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/webdata_healthgateway_downloaded_reports_monthly_20240507T175519_part000.csv

Objects to process: 1 Objects processed to s3 /good: 1

List of objects processed to S3 /good: 1: processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/webdata_healthgateway_downloaded_reports_monthly_20240507T175519_part000


3. Check to see if the file appear in the s3 processed batch bucket: 

- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/app_ratings_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/app_ratings_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_cards_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_cards_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_reports_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/&showversions=false

4. Check to see if the file appear in the s3 client bucket: 

- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/app_ratings_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/app_ratings_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_cards_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_cards_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_reports_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/&showversions=false

5. Check to see if the file appear in the s3 processed good bucket: 

- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/app_ratings_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/app_ratings_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_cards_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_cards_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_reports_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/&showversions=false