Creates a proof of concept microservice that will transfer data out of a MySQL database and store the resulting file in S3.
The microservice uses a config file full of settings to specify a query to run in the MySQL database. The python script will then take the results of that query and store them in a file that is placed into S3.
Testing instructions:
Log into the ec2 instance through the following commands
awsmfa prod <AWS OTP>
microservice_ssm
cd /home/microservice/branch/GDXDSD-5704_mysql_to_s3/mysql_to_s3
Run the following command and compare its output to what's expected.
pipenv run python mysql_to_s3.py -c config.d/example.json
Objects to store: 1
Objects stored to s3 /client: 1
List of objects stored to S3 /client:
1: client/doug_test/GDXDSD-5704/test_mysql-to-s3_20240620T172012.txt
Objects to process: 1
Objects processed to s3 /good: 1
List of objects processed to S3 /good:
1: processed/good/client/doug_test/GDXDSD-5704/test_mysql-to-s3_20240620T172012
3. Check to see if the file appear in the s3 processed batch bucket:
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/batch/client/test_microservice-iam-key/mysql_to_s3/&showversions=false
4. Check to see if the file appear in the s3 client bucket:
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/test_microservice-iam-key/mysql_to_s3/&showversions=false
5. Check to see if the file appear in the s3 processed good bucket:
- https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=processed/good/client/test_microservice-iam-key/mysql_to_s3/&showversions=false
6. Review the file generated and placed in the [client folder](https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/test_microservice-iam-key/mysql_to_s3/&showversions=false) to make sure it looks like a working pipe separated value file
This PR does the following:
Creates a proof of concept microservice that will transfer data out of a MySQL database and store the resulting file in S3.
The microservice uses a config file full of settings to specify a query to run in the MySQL database. The python script will then take the results of that query and store them in a file that is placed into S3.
Testing instructions:
Report: mysql_to_s3.py
Config: config.d/example.json
DML: example.sql
Microservice started at: 2024-06-20 10:20:11-0700 (PDT), ended at: 2024-06-20 10:20:12-0700 (PDT), elapsing: 0:00:00.863559.
Objects loaded to S3 /batch: 1/1 Objects successfully loaded to S3 /batch: 1
List of objects successfully loaded to S3 /batch
Objects to store: 1 Objects stored to s3 /client: 1
List of objects stored to S3 /client: 1: client/doug_test/GDXDSD-5704/test_mysql-to-s3_20240620T172012.txt
Objects to process: 1 Objects processed to s3 /good: 1
List of objects processed to S3 /good: 1: processed/good/client/doug_test/GDXDSD-5704/test_mysql-to-s3_20240620T172012