Objects to store: 1
Objects stored to s3 /client: 1
List of objects stored to S3 /client:
1: client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/webdata_healthgateway_downloaded_reports_monthly_20240507T175519_part000.csv
Objects to process: 1
Objects processed to s3 /good: 1
List of objects processed to S3 /good:
1: processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/webdata_healthgateway_downloaded_reports_monthly_20240507T175519_part000
3. Check to see if the file appear in the s3 processed batch bucket:
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/app_ratings_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/app_ratings_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_cards_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_cards_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_reports_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/&showversions=false
4. Check to see if the file appear in the s3 client bucket:
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/app_ratings_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/app_ratings_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_cards_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_cards_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_reports_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/&showversions=false
5. Check to see if the file appear in the s3 processed good bucket:
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/app_ratings_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/app_ratings_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_cards_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_cards_monthly/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/daily/downloaded_reports_daily/v01/&showversions=false
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/&showversions=false
This PR does the following:
Updates the following DMLs to use the newly updated stable name PDTs :
Please note that some of the config file were modified so that the tests can run. The modified configs will be added to the ticket in a zip file.
Testing instructions:
Report: redshift_to_s3.py
Config: config.d/webdata_healthgateway_downloaded_reports_monthly.json
DML: webdata_healthgateway_downloaded_reports_monthly.sql
Microservice started at: 2024-05-07 10:55:19-0700 (PDT), ended at: 2024-05-07 10:55:20-0700 (PDT), elapsing: 0:00:01.073265.
Objects loaded to S3 /batch: 1/1 Objects successfully loaded to S3 /batch: 1
List of objects successfully loaded to S3 /batch
Objects to store: 1 Objects stored to s3 /client: 1
List of objects stored to S3 /client: 1: client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/webdata_healthgateway_downloaded_reports_monthly_20240507T175519_part000.csv
Objects to process: 1 Objects processed to s3 /good: 1
List of objects processed to S3 /good: 1: processed/good/client/doug_test/GDXDSD-6969/webdata_healthgateway/new/monthly/downloaded_reports_monthly/v01/webdata_healthgateway_downloaded_reports_monthly_20240507T175519_part000