search

bcgov / GDX-Analytics-microservice

The collection of GDX-Analytics Python microservices used to load and process data between systems and services.
Apache License 2.0
2 stars 0 forks source link

Gdxdsd 5650 s3 to sfts error 2023 02 28 #167

Closed doughon closed 1 year ago

doughon commented 1 year ago

This PR does the following:

Addresses an error that caused previously processes objects in an S3 subfolder not specified in the config to be recognized as a object to be uploaded to SFTS.

Testing these changes requires modifications to the existing config files that are not a part of the PR. You can find these modified config files attached to the Jira ticket.

Background for testing:

The following is a link to where in S3 that s3_to_sfts.py looks for for files to process: https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?prefix=client/doug_test/GDXDSD-5650/pmrp_gdx/pmrp_date_range/&region=ca-central-1

It will look for these files in

And if it doesn't find the files it will attempt to upload the file into SFTS

For testing purposes, we are looking to see that pmrp_20200328_20200328_20200331T143401_part000 is selected for processing and transferred to SFTS And that the following are not flagged to be processed

Testing instructions:

  1. Log into the ec2 instance through the following commands 
    awsmfa prod <AWS OTP>
microservice_ssm
cd /home/microservice/branch/GDXDSD-5650-s3_to_sfts-error-2023-02-28/s3_to_sfts
  2. Run the following command and compare its output to what's expected. 
    pipenv run python s3_to_sfts.py -c config.d/pmrp_date_range.json
    
Report: s3_to_sfts.py

Config: config.d/pmrp_date_range.json

Microservice started at: 2023-03-01 14:51:51-0800 (PST), ended at: 2023-03-01 14:51:56-0800 (PST), elapsing: 0:00:04.488708.

Items to process: 1 Objects successfully processed to s3: 1 Objects unsuccessfully processed to s3: 0 Objects successfully processed to sfts: 1

Objects loaded to S3 /good:

1: processed/good/client/doug_test/GDXDSD-5650/google-mybusiness-sfts_sbc/pmrp_date_range/pmrp_20200328_20200328_20200331T143401_part000

3. Run the following command to check the logs to see if it identified the files for processing correctly

vim logs/s3_to_sfts.log

INFO:botocore.credentials:2023-03-01 22:51:51,810:Found credentials in environment variables. INFO:main:2023-03-01 22:51:51,991:pmrp_20200625_20200710_20200714T173645_part000 was processed as good already. INFO:main:2023-03-01 22:51:52,003:pmrp_20201103_20201114_20201126T183404_part000 was processed as good already. INFO:main:2023-03-01 22:51:52,013:pmrp_20180929_20200327_20200331T005636_part000 was processed as good already. INFO:main:2023-03-01 22:51:52,039:pmrp_20200328_20200328_20200331T143401_part000 has not been processed. INFO:main:2023-03-01 22:51:52,039:added 'pmrp_20200328_20200328_20200331T143401_part000' for processing INFO:main:2023-03-01 22:51:52,083:file for xfer -s call is /home/microservice/branch/GDXDSD-5650-s3_to_sfts-error-2023-02-28/s3_to_sfts/tmp/sfst_conf INFO:main:2023-03-01 22:51:52,083:Contents: INFO:main:2023-03-01 22:51:52,083:cd CITZ_AP_GDX/Shared/microservice/doug_test/GDXDSD-5650/SBC/Shared/Secured/SS_BI/SS_GDX_IN INFO:main:2023-03-01 22:51:52,083:put ./tmp/pmrp_20200328_20200328_20200331T143401_part000.csv INFO:main:2023-03-01 22:51:52,083:quit INFO:main:2023-03-01 22:51:52,083: INFO:main:2023-03-01 22:51:52,083:trying to call subprocess: INFO:main:2023-03-01 22:51:52,083:xfer.jar: /home/microservice/MOVEit-Xfer//xfer.jar INFO:main:2023-03-01 22:51:52,083:jna.jar : /home/microservice/MOVEit-Xfer//jna.jar INFO:main:2023-03-01 22:51:56,235:User GDXSBCTR signed on INFO:main:2023-03-01 22:51:56,235:--> cd CITZ_AP_GDX/Shared/microservice/doug_test/GDXDSD-5650/SBC/Shared/Secured/SS_BI/SS_GDX_IN INFO:main:2023-03-01 22:51:56,235:--> put ./tmp/pmrp_20200328_20200328_20200331T143401_part000.csv INFO:main:2023-03-01 22:51:56,235:Upload complete. File ID 939342992 INFO:main:2023-03-01 22:51:56,235:5380 bytes sent in 2.12 seconds at 2.48 kB/s INFO:main:2023-03-01 22:51:56,235:--> quit INFO:main:2023-03-01 22:51:56,235: INFO:main:2023-03-01 22:51:56,278:copied client/doug_test/GDXDSD-5650/pmrp_gdx/pmrp_date_range/pmrp_20200328_20200328_20200331T143401_part000 to processed/good/client/doug_test/GDXDSD-5650/google-mybusiness-sfts_sbc/pmrp_date_range/pmrp_20200328_20200328_20200331T143401_part000 INFO:main:2023-03-01 22:51:56,278:Exiting with code 0 : Finished successfully.


5. Check to see if the file appear in the s3 processed good bucket: https://s3.console.aws.amazon.com/s3/buckets/sp-ca-bc-gov-131565110619-12-microservices?prefix=processed/good/client/doug_test/GDXDSD-5650/google-mybusiness-sfts_sbc/pmrp_date_range/&region=ca-central-1
6. Check to see if the file appear in SFTS: https://filetransfer.gov.bc.ca/human.aspx?r=90096197&arg06=939242669&arg12=filelist