search

bcgov / GDX-Analytics-microservice

The collection of GDX-Analytics Python microservices used to load and process data between systems and services.
Apache License 2.0
2 stars 0 forks source link

Gdxdsd 7112 looker dashboard usage sqlalchemy #228

Closed doughon closed 2 months ago

doughon commented 2 months ago

This PR does the following:

  1. Changes the connection in the looker_dashboard_usage microservice to use and SQLAlchemy engine
  2. Changes the DBAPI from using mysql-connector-python to PyMySQL
  3. As a consequence, updated the dashboard.history query to escape the '%' character

Testing the changes in this microservice requires a file to be modified. This test file will be attached to the ticket

File review:

  1. Review the virtual environment files to make sure SQLAlchemy and PyMySQL is installed

Logging into the EC2 testing environment:

  1. Log into the ec2 instance through the following commands 
    $ awsmfa prod ######
Generating new IAM STS Token ...
STS Session Token generated and saved in profile mfa successfully.
$ microservice_ssm
$ cd /home/microservice/branch/GDXDSD-7112_looker_dashboard_usage_sqlalchemy/looker_dashboard_usage

    Testing the looker_dashboard_usage.py script

  2. Run the following command to test the looker_dashboard_usage microservice and compare its output to what's expected. 
    $ /home/microservice/.local/bin/pipenv run python3.8 looker_dashboard_usage.py looker_dashboard_usage.json 
    
Report: looker_dashboard_usage.py

Config: looker_dashboard_usage.json

This microservice ran: successful

Microservice started at: 2024-08-26 16:55:56-0700 (PDT), ended at: 2024-08-26 16:55:57-0700 (PDT), elapsing: 0:00:01.535820.

Objects to process: 4 Objects that failed to process: 0 Objects output to 'processed/good': 4 Objects output to 'processed/bad': 0

List of objects successfully fully ingested from S3, processed, loaded to S3 ('good'), and copied to Redshift:

  1. client/doug_test/GDXDSD-7112/looker_dashboards/dashboard.2024-08-25

  2. client/doug_test/GDXDSD-7112/looker_dashboards/history.2024-08-25

  3. client/doug_test/GDXDSD-7112/looker_dashboards/user.2024-08-25

  4. client/doug_test/GDXDSD-7112/looker_dashboards/looker_user_facts.2024-08-25

  5. Check to see if the four new files appear in the s3 client bucket: https://ca-central-1.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?region=ca-central-1&bucketType=general&prefix=client/doug_test/GDXDSD-7112/looker_dashboards/&showversions=false