Integration of fac-backup-utility

asteel-gsa commented 1 month ago

Description:

Integration of the fac-backup-utility within the application. This, at a high level, removes the previously used django-dbbackup and replaces it with a table by table dump of the rds to s3 and secondary rds.

Note:

It appears my IDE stripped some line endings, hence why some of the files look like they have been substantially changed, but I will list everything below.

Overview:

dev, staging, and preview have been given their own, dedicated s3 bucket backups, managed by their respective module.
prod still has its backups bucket managed by the meta module.
all environments receive a new rds fac-snapshot-db.
meta managed production backups get tagged with s3 and are now up to date on version.
manifest.yml has been modified to ensure that backups and fac-snapshot-db are now bound to the running app, and will not collide due to sharing across spaces.
terraform-apply-env.yml no longer shares the production backups to staging.
docker-compose.yml and docker-compose-web.yml have been updated to include a new database so this can be run locally.
two bash scripts, fac-backup-util.sh and fac-restore-util.sh have been added and will replace the existing bash scripts, i.e backup-database.yml.
workflows have been modified to use the new backup/restore system.
only fac-admins can run the restore process at this time via workflow.
a new, scheduled job, which uses a matrix, has been added, which will do a scheduled_backup operation on all 3 environments via matrix.

Common Commands:

### Common Commands:
./fac-backup-util.sh v0.1.2 initial_backup
# Curl the utility
# Install AWS
# DB to S3 table dump (backups)
# DB to DB table dump (fac-db -> fac-snapshot-db) [No Truncate, as tables dont exist]
# AWS S3 sync (fac-private-s3 -> backups)

./fac-backup-util.sh v0.1.2 deploy_backup
# Curl the utility
# Install AWS
# DB to S3 table dump (backups)
# AWS S3 sync (fac-private-s3 -> backups)

./fac-backup-util.sh v0.1.2 scheduled_backup
# Curl the utility
# Install AWS
# DB to S3 table dump (fac-db -> backups)
# DB to DB table dump (fac-db -> fac-snapshot-db) [Truncate target table before dump]
# AWS S3 sync (fac-private-s3 -> backups)

./fac-restore-util.sh v0.1.2 s3_restore 202405301232
# Curl the utility
# Install AWS
# DB to S3 table dump (backups -> fac-db) [Truncate target table before --data-only pg_restore]
# AWS S3 sync (backups -> fac-private-s3)

./fac-restore-util.sh v0.1.2 db_restore
# Curl the utility
# Install AWS
# DB to DB table dump (fac-snapshot-db -> fac-db) [Truncate target table before dump]
# AWS S3 sync (fac-private-s3 -> backups)

Matrix Test/Validation: Run: here

Todo:

review fac-backup-util main repo

Prep work before merge:

Unshare backups from prod manually: cf unshare-service backups -s staging -f
clean and delete backups in dev so terraform can rebuild it

github-actions[bot] commented 1 month ago

File	Coverage		Missing
All files	`87%`	:white_check_mark:
api/serializers.py	`88%`	:white_check_mark:	`177-178` `183` `188`
api/test_views.py	`95%`	:white_check_mark:	`103`
api/uei.py	`88%`	:white_check_mark:	`87` `118-119` `163` `167-168`
api/views.py	`98%`	:white_check_mark:	`198-199` `337-338`
audit/forms.py	`60%`	:x:	`31-38` `109-116`
audit/intake_to_dissemination.py	`88%`	:white_check_mark:	`57-62` `264` `308-316`
audit/mixins.py	`96%`	:white_check_mark:	`28`
audit/test_commands.py	`91%`	:white_check_mark:
audit/test_intakelib.py	`88%`	:white_check_mark:	`154-158`
audit/test_manage_submission_access_view.py	`98%`	:white_check_mark:	`15` `19`
audit/test_mixins.py	`90%`	:white_check_mark:	`159-160` `164-166` `254-255` `259-261`
audit/test_validators.py	`95%`	:white_check_mark:	`439` `443` `611-612` `851` `858` `865` `872` `1117-1118` `1149-1150` `1175-1180`
audit/test_views.py	`98%`	:white_check_mark:	`132`
audit/test_workbooks_should_fail.py	`88%`	:white_check_mark:	`58` `87-88` `92`
audit/test_workbooks_should_pass.py	`87%`	:white_check_mark:	`59` `74-76`
audit/utils.py	`86%`	:white_check_mark:	`9` `19` `60-62` `65`
audit/validators.py	`93%`	:white_check_mark:	`138` `190` `279` `419-420` `435-436` `519-520` `622-626` `631-635` `651-660`
audit/cross_validation/additional_ueis.py	`93%`	:white_check_mark:	`33`
audit/cross_validation/check_award_ref_declaration.py	`90%`	:white_check_mark:
audit/cross_validation/check_award_reference_uniqueness.py	`93%`	:white_check_mark:
audit/cross_validation/check_certifying_contacts.py	`87%`	:white_check_mark:
audit/cross_validation/check_findings_count_consistency.py	`87%`	:white_check_mark:	`35`
audit/cross_validation/check_ref_number_in_cap.py	`91%`	:white_check_mark:
audit/cross_validation/check_ref_number_in_findings_text.py	`91%`	:white_check_mark:
audit/cross_validation/errors.py	`78%`	:x:	`30` `77`
audit/cross_validation/naming.py	`93%`	:white_check_mark:	`229`
audit/cross_validation/submission_progress_check.py	`91%`	:white_check_mark:	`83` `126` `174` `182-183`
audit/cross_validation/tribal_data_sharing_consent.py	`81%`	:x:	`33` `36` `40`
audit/cross_validation/validate_general_information.py	`65%`	:x:	`77` `81-84` `96` `99`
audit/fixtures/dissemination.py	`71%`	:x:	`38`
audit/fixtures/single_audit_checklist.py	`55%`	:x:	`160-197` `245-254`
audit/intakelib/exceptions.py	`71%`	:x:	`7-9` `12`
audit/intakelib/intermediate_representation.py	`91%`	:white_check_mark:	`27-28` `73` `91` `129` `200-203` `212-213` `283-284`
audit/intakelib/mapping_audit_findings.py	`97%`	:white_check_mark:	`55`
audit/intakelib/mapping_audit_findings_text.py	`97%`	:white_check_mark:	`54`
audit/intakelib/mapping_federal_awards.py	`93%`	:white_check_mark:	`92`
audit/intakelib/mapping_util.py	`79%`	:x:	`21` `25` `29` `63` `99` `104-105` `114-120` `130` `145` `150`
audit/intakelib/checks/check_all_unique_award_numbers.py	`79%`	:x:	`24`
audit/intakelib/checks/check_cluster_names.py	`75%`	:x:	`20-25`
audit/intakelib/checks/check_cluster_total.py	`95%`	:white_check_mark:	`99`
audit/intakelib/checks/check_finding_reference_pattern.py	`74%`	:x:	`34` `44-45`
audit/intakelib/checks/check_findings_grid_validation.py	`89%`	:white_check_mark:	`59`
audit/intakelib/checks/check_has_all_the_named_ranges.py	`95%`	:white_check_mark:	`66`
audit/intakelib/checks/check_is_a_workbook.py	`69%`	:x:	`20`
audit/intakelib/checks/check_loan_balance_entries.py	`83%`	:x:	`28`
audit/intakelib/checks/check_look_for_empty_rows.py	`91%`	:white_check_mark:	`18`
audit/intakelib/checks/check_no_major_program_no_type.py	`76%`	:x:	`18` `27`
audit/intakelib/checks/check_no_repeat_findings.py	`88%`	:white_check_mark:	`21`
audit/intakelib/checks/check_other_cluster_names.py	`81%`	:x:	`23` `33`
audit/intakelib/checks/check_passthrough_name_when_no_direct.py	`83%`	:x:	`11` `49` `58`
audit/intakelib/checks/check_sequential_award_numbers.py	`82%`	:x:	`25` `35`
audit/intakelib/checks/check_start_and_end_rows_of_all_columns_are_same.py	`89%`	:white_check_mark:	`14`
audit/intakelib/checks/check_state_cluster_names.py	`81%`	:x:	`23` `33`
audit/intakelib/checks/check_version_number.py	`73%`	:x:	`30` `40-41`
audit/intakelib/checks/runners.py	`95%`	:white_check_mark:	`187` `217`
audit/intakelib/common/util.py	`90%`	:white_check_mark:	`22` `39`
audit/intakelib/transforms/xform_rename_additional_notes_sheet.py	`81%`	:x:	`14`
audit/management/commands/load_fixtures.py	`47%`	:x:	`40-46`
audit/models/models.py	`85%`	:white_check_mark:	`58` `60` `65` `67` `209` `215` `227` `239-242` `260` `437` `455-456` `464` `486` `584-585` `589` `597` `606` `612`
audit/views/audit_info_form_view.py	`27%`	:x:	`25-74` `77-117` `120-137`
audit/views/manage_submission.py	`86%`	:white_check_mark:	`73-80`
audit/views/manage_submission_access.py	`98%`	:white_check_mark:	`113-114`
audit/views/pre_dissemination_download_view.py	`78%`	:x:	`15-16` `21-22` `29-39`
audit/views/submission_progress_view.py	`89%`	:white_check_mark:	`117` `182-183`
audit/views/tribal_data_consent.py	`34%`	:x:	`23-41` `44-79`
audit/views/unlock_after_certification.py	`57%`	:x:	`28-51` `73-87`
audit/views/upload_report_view.py	`26%`	:x:	`32-35` `44` `91-117` `120-170` `178-209`
audit/views/views.py	`53%`	:x:	`74` `81-100` `123-124` `198-199` `220-230` `257` `268-269` `280-281` `283-287` `329-342` `345-359` `364-377` `394-400` `405-425` `452-456` `461-490` `533-537` `542-562` `589-593` `598-627` `670-674` `679-691` `694-704` `709-721` `754-768`
census_historical_migration/change_record.py	`98%`	:white_check_mark:	`30`
census_historical_migration/end_to_end_core.py	`26%`	:x:	`57-89` `93-111` `116-155` `161-187` `246-258` `263` `273-307`
census_historical_migration/invalid_record.py	`94%`	:white_check_mark:	`50` `54` `58` `62` `66`
census_historical_migration/migration_result.py	`75%`	:x:	`17` `21` `25` `29` `33-42` `46`
census_historical_migration/report_type_flag.py	`96%`	:white_check_mark:	`19`
census_historical_migration/test_federal_awards_xforms.py	`99%`	:white_check_mark:	`219-220`
census_historical_migration/sac_general_lib/audit_information.py	`91%`	:white_check_mark:	`28` `82-87` `336`
census_historical_migration/sac_general_lib/cognizant_oversight.py	`68%`	:x:	`11`
census_historical_migration/sac_general_lib/general_information.py	`86%`	:white_check_mark:	`166-167` `177-178` `186-187` `195-200` `233-255` `354-355`
census_historical_migration/sac_general_lib/sac_creator.py	`90%`	:white_check_mark:	`34`
census_historical_migration/sac_general_lib/utils.py	`84%`	:x:	`35` `62-71`
census_historical_migration/transforms/xform_remove_hyphen_and_pad_zip.py	`92%`	:white_check_mark:	`18`
census_historical_migration/transforms/xform_retrieve_uei.py	`67%`	:x:	`10`
census_historical_migration/transforms/xform_string_to_bool.py	`87%`	:white_check_mark:	`17`
census_historical_migration/workbooklib/additional_eins.py	`84%`	:x:	`58-60` `67-77`
census_historical_migration/workbooklib/additional_ueis.py	`77%`	:x:	`27-29` `36-46`
census_historical_migration/workbooklib/corrective_action_plan.py	`46%`	:x:	`49-51` `65` `93-125` `134-153`
census_historical_migration/workbooklib/excel_creation_utils.py	`69%`	:x:	`110` `119-124` `129-136` `140-158` `171-175` `189-192`
census_historical_migration/workbooklib/federal_awards.py	`77%`	:x:	`181-184` `262-301` `487` `554-562` `572-597` `621-622` `918-1022`
census_historical_migration/workbooklib/findings.py	`69%`	:x:	`81-86` `154-174` `179-199` `216-218` `330-354`
census_historical_migration/workbooklib/findings_text.py	`46%`	:x:	`50-51` `67` `97-129` `138-160`
census_historical_migration/workbooklib/notes_to_sefa.py	`66%`	:x:	`34-38` `101-102` `104-105` `107-108` `136-143` `152-160` `168-170` `279-326`
census_historical_migration/workbooklib/post_upload_utils.py	`21%`	:x:	`22-35` `66-83` `89-111`
census_historical_migration/workbooklib/secondary_auditors.py	`88%`	:white_check_mark:	`128-130` `186-205`
census_historical_migration/workbooklib/workbook_builder.py	`38%`	:x:	`14-17` `26-41`
census_historical_migration/workbooklib/workbook_builder_loader.py	`41%`	:x:	`18-30`
config/error_handlers.py	`94%`	:white_check_mark:	`22`
config/test_settings.py	`92%`	:white_check_mark:	`33-34` `49-50`
config/urls.py	`72%`	:x:	`88`
dissemination/file_downloads.py	`81%`	:x:	`43-61` `91-93`
dissemination/forms.py	`87%`	:white_check_mark:	`135` `144` `255`
dissemination/search.py	`88%`	:white_check_mark:	`113` `115` `119` `127-128`
dissemination/summary_reports.py	`78%`	:x:	`274` `300-302` `306-310` `421` `438` `459` `511-575` `603` `638-640` `664-672`
dissemination/test_search.py	`93%`	:white_check_mark:	`51-66` `473-474` `579-596` `608-632` `644-669` `677-693`
dissemination/test_summary_reports.py	`98%`	:white_check_mark:
dissemination/views.py	`75%`	:x:	`134` `140-142` `159-225` `268` `298` `300` `336` `387` `389` `391` `469-474`
dissemination/migrations/0002_general_fac_accepted_date.py	`47%`	:x:	`10-12`
dissemination/searchlib/search_alns.py	`37%`	:x:	`44-58` `78-110` `115-177` `184-187`
dissemination/searchlib/search_direct_funding.py	`86%`	:white_check_mark:	`21-22`
dissemination/searchlib/search_findings.py	`76%`	:x:	`18-24` `34` `36` `38`
dissemination/searchlib/search_general.py	`96%`	:white_check_mark:	`138`
dissemination/searchlib/search_passthrough_name.py	`35%`	:x:	`21-31`
djangooidc/backends.py	`78%`	:x:	`32` `57-63`
djangooidc/exceptions.py	`66%`	:x:	`19` `21` `23` `28`
djangooidc/oidc.py	`16%`	:x:	`32-35` `45-51` `64-70` `92-149` `153-199` `203-226` `230-275` `280-281` `286`
djangooidc/views.py	`81%`	:x:	`22` `43` `109-110` `117`
djangooidc/tests/common.py	`97%`	:white_check_mark:
report_submission/forms.py	`92%`	:white_check_mark:	`35`
report_submission/test_views.py	`98%`	:white_check_mark:	`835`
report_submission/views.py	`79%`	:x:	`97` `250` `274-275` `280-281` `321-491` `494-504` `563` `599-601` `609-610` `613-615`
report_submission/templatetags/get_attr.py	`76%`	:x:	`8` `11-14` `18`
support/admin.py	`88%`	:white_check_mark:	`76` `79` `84` `91-97` `100-102`
support/cog_over.py	`91%`	:white_check_mark:	`29-32` `92` `116-120` `156`
support/test_admin_api.py	`81%`	:x:	`23` `147-148` `237-238` `317-318`
support/test_cog_over.py	`98%`	:white_check_mark:	`174-175` `264`
support/management/commands/seed_cog_baseline.py	`98%`	:white_check_mark:	`20-21`
support/models/cog_over.py	`89%`	:white_check_mark:	`103-104`
tools/update_program_data.py	`89%`	:white_check_mark:	`96`
users/admin.py	`99%`	:white_check_mark:	`27`
users/auth.py	`96%`	:white_check_mark:	`58-59`
users/models.py	`96%`	:white_check_mark:	`18` `74-75`
users/fixtures/user_fixtures.py	`91%`	:white_check_mark:

Minimum allowed coverage is 85%

Generated by :monkey: cobertura-action against f69ad62ba39ea7490aaf286d720f74636dad72fb

asteel-gsa commented 1 month ago

Notes from review w/ Jadud:

~change 18f -> gsa-tts in terraform~
~consider daily backups~ (just needs a new workflow)
~file structure for s3 (hourly-00, hourly-02, etc etc and daily-0, daily-1)~
~deploy backup should go to rds instead of s3, and save the s3 for scheduled backups to maintain point in time recovery over more ranges, and utilize the rds backup methodology as a true cold store for deployment backups~

Action items from data team discussion:

s3_to_db pipe rather than use disk. (This is probably a task @jadudm and I will have to take some time on and diagnose for the next iteration of development)
~PG_DUMP_TABLE() format c -> plain for db_to_db dumps~

asteel-gsa commented 1 month ago

Closing this temporarily, as I will be re-testing the workflows with push.

asteel-gsa commented 1 month ago

_deploybackup: Workflow Run Operations: db to db backup, aws s3 sync Confirmation: Tunneled into fac-snapshot-db and confirmed tables existing. AWS S3 Sync passed as there has been no change of data in either bucket.

_scheduledbackup: Workflow Run Operations: db to s3 backup, aws s3 sync s3 folder: scheduled/06-04-18/

# stored in scheduled/mm-dd-HH/ format
Alex Steel@DESKTOP-NL4DO24 MINGW64 ~/Code/FAC (as/db-backup-util)
$ aws s3 ls s3://$BUCKET/backups/scheduled/06-04-18/
2024-06-04 14:31:20       5037 public-audit_access.dump
2024-06-04 14:33:02       5595 public-audit_deletedaccess.dump
2024-06-04 14:31:20       4346 public-audit_excelfile.dump
2024-06-04 14:32:50       5649 public-audit_singleauditchecklist.dump
2024-06-04 14:31:38       4754 public-audit_singleauditreportfile.dump
.........

_dailybackup: Workflow Run Operations: db to s3 backup, aws s3 sync s3 folder: daily/06-04/

# stored in daily/mm-dd/ format
Alex Steel@DESKTOP-NL4DO24 MINGW64 ~/Code/FAC (as/db-backup-util)
$ aws s3 ls s3://$BUCKET/backups/daily/06-04/
2024-06-04 14:41:32       5037 public-audit_access.dump
2024-06-04 14:42:55       5595 public-audit_deletedaccess.dump
2024-06-04 14:42:54       4346 public-audit_excelfile.dump
2024-06-04 14:39:41       5649 public-audit_singleauditchecklist.dump
2024-06-04 14:39:51       4754 public-audit_singleauditreportfile.dump
.........

_s3restore: Workflow Run Operations: s3 to db restore, aws s3 sync s3 folder: `daily/06-04/

# Example output of a specific table
2024-06-04T14:51:06.56-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:06 S3 Copying cg-6a89718b-e7b6-4f63-a2ae-78063df394e7 to local disk.  
   2024-06-04T14:51:06.56-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:06 Running command: /home/vcap/app/bin/aws s3 cp s3://cg-6a89718b-e7b6-4f63-a2ae-78063df394e7/backups/daily/06-04/public-census_historical_migration_elecnotes.dump ./pg_dump_tables/
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:07 Completed 256.0 KiB/38.9 MiB (2.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 512.0 KiB/38.9 MiB (5.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 768.0 KiB/38.9 MiB (7.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.0 MiB/38.9 MiB (10.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.2 MiB/38.9 MiB (12.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.5 MiB/38.9 MiB (14.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.8 MiB/38.9 MiB (16.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.0 MiB/38.9 MiB (18.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.2 MiB/38.9 MiB (20.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.5 MiB/38.9 MiB (22.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.8 MiB/38.9 MiB (24.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.0 MiB/38.9 MiB (26.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.2 MiB/38.9 MiB (28.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.5 MiB/38.9 MiB (30.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.8 MiB/38.9 MiB (32.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.0 MiB/38.9 MiB (34.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.2 MiB/38.9 MiB (35.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.5 MiB/38.9 MiB (37.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.8 MiB/38.9 MiB (38.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.0 MiB/38.9 MiB (40.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.2 MiB/38.9 MiB (42.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.5 MiB/38.9 MiB (43.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.8 MiB/38.9 MiB (45.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.0 MiB/38.9 MiB (47.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.2 MiB/38.9 MiB (48.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.5 MiB/38.9 MiB (50.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.8 MiB/38.9 MiB (51.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.0 MiB/38.9 MiB (52.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.2 MiB/38.9 MiB (54.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.5 MiB/38.9 MiB (55.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.8 MiB/38.9 MiB (57.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.0 MiB/38.9 MiB (58.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.2 MiB/38.9 MiB (59.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.5 MiB/38.9 MiB (60.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.8 MiB/38.9 MiB (62.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.0 MiB/38.9 MiB (63.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.2 MiB/38.9 MiB (65.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.5 MiB/38.9 MiB (65.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.8 MiB/38.9 MiB (67.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.0 MiB/38.9 MiB (68.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.2 MiB/38.9 MiB (69.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.5 MiB/38.9 MiB (71.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.8 MiB/38.9 MiB (71.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.0 MiB/38.9 MiB (73.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.2 MiB/38.9 MiB (74.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.5 MiB/38.9 MiB (75.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.8 MiB/38.9 MiB (76.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.0 MiB/38.9 MiB (77.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.2 MiB/38.9 MiB (79.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.5 MiB/38.9 MiB (79.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.8 MiB/38.9 MiB (80.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.0 MiB/38.9 MiB (81.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.2 MiB/38.9 MiB (82.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.5 MiB/38.9 MiB (84.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.8 MiB/38.9 MiB (84.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.0 MiB/38.9 MiB (86.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.2 MiB/38.9 MiB (87.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.5 MiB/38.9 MiB (87.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.8 MiB/38.9 MiB (89.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.0 MiB/38.9 MiB (89.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.2 MiB/38.9 MiB (90.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.5 MiB/38.9 MiB (91.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.8 MiB/38.9 MiB (92.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.0 MiB/38.9 MiB (94.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.2 MiB/38.9 MiB (94.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.5 MiB/38.9 MiB (95.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.8 MiB/38.9 MiB (96.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.0 MiB/38.9 MiB (97.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.2 MiB/38.9 MiB (98.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.5 MiB/38.9 MiB (98.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.8 MiB/38.9 MiB (99.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.0 MiB/38.9 MiB (100.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.2 MiB/38.9 MiB (101.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.5 MiB/38.9 MiB (102.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.8 MiB/38.9 MiB (102.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.0 MiB/38.9 MiB (103.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.2 MiB/38.9 MiB (104.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.5 MiB/38.9 MiB (105.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.8 MiB/38.9 MiB (106.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.0 MiB/38.9 MiB (106.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.2 MiB/38.9 MiB (107.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.5 MiB/38.9 MiB (107.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.8 MiB/38.9 MiB (108.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.0 MiB/38.9 MiB (109.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.2 MiB/38.9 MiB (109.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.5 MiB/38.9 MiB (110.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.8 MiB/38.9 MiB (110.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.0 MiB/38.9 MiB (111.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.2 MiB/38.9 MiB (111.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.5 MiB/38.9 MiB (112.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.8 MiB/38.9 MiB (113.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.0 MiB/38.9 MiB (114.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.2 MiB/38.9 MiB (115.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.5 MiB/38.9 MiB (115.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.8 MiB/38.9 MiB (116.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.0 MiB/38.9 MiB (117.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.2 MiB/38.9 MiB (118.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.5 MiB/38.9 MiB (119.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.8 MiB/38.9 MiB (118.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.0 MiB/38.9 MiB (118.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.2 MiB/38.9 MiB (119.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.5 MiB/38.9 MiB (120.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.8 MiB/38.9 MiB (120.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.0 MiB/38.9 MiB (120.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.2 MiB/38.9 MiB (121.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.5 MiB/38.9 MiB (122.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.8 MiB/38.9 MiB (122.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.0 MiB/38.9 MiB (123.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.2 MiB/38.9 MiB (124.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.5 MiB/38.9 MiB (124.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.8 MiB/38.9 MiB (125.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.9 MiB/38.9 MiB (125.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.1 MiB/38.9 MiB (125.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.4 MiB/38.9 MiB (126.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.6 MiB/38.9 MiB (127.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.9 MiB/38.9 MiB (127.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.1 MiB/38.9 MiB (128.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.4 MiB/38.9 MiB (129.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.6 MiB/38.9 MiB (129.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.9 MiB/38.9 MiB (130.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.1 MiB/38.9 MiB (130.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.4 MiB/38.9 MiB (131.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.6 MiB/38.9 MiB (131.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.9 MiB/38.9 MiB (132.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.1 MiB/38.9 MiB (133.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.4 MiB/38.9 MiB (134.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.6 MiB/38.9 MiB (133.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.9 MiB/38.9 MiB (133.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.1 MiB/38.9 MiB (134.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.4 MiB/38.9 MiB (135.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.6 MiB/38.9 MiB (134.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.9 MiB/38.9 MiB (135.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.1 MiB/38.9 MiB (136.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.4 MiB/38.9 MiB (136.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.6 MiB/38.9 MiB (136.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.9 MiB/38.9 MiB (137.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.1 MiB/38.9 MiB (138.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.4 MiB/38.9 MiB (138.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.6 MiB/38.9 MiB (138.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.9 MiB/38.9 MiB (139.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.1 MiB/38.9 MiB (139.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.4 MiB/38.9 MiB (139.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.6 MiB/38.9 MiB (139.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.9 MiB/38.9 MiB (140.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.1 MiB/38.9 MiB (141.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.4 MiB/38.9 MiB (141.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.6 MiB/38.9 MiB (141.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.9 MiB/38.9 MiB (142.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.1 MiB/38.9 MiB (143.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.4 MiB/38.9 MiB (143.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.6 MiB/38.9 MiB (144.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.9 MiB/38.9 MiB (144.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.1 MiB/38.9 MiB (145.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.4 MiB/38.9 MiB (145.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.6 MiB/38.9 MiB (146.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.9 MiB/38.9 MiB (146.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT download: s3://cg-6a89718b-e7b6-4f63-a2ae-78063df394e7/backups/daily/06-04/public-census_historical_migration_elecnotes.dump to pg_dump_tables/public-census_historical_migration_elecnotes.dump
   2024-06-04T14:51:08.43-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:08 TRUNCATE truncating table census_historical_migration_elecnotes    
   2024-06-04T14:51:08.43-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:08 /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/psql running       
   2024-06-04T14:51:08.47-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:08 RESTORE /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/pg_restore targeting public.census_historical_migration_elecnotes
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:18 pg_restore: connecting to database for restore
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT pg_restore: processing data for table "public.census_historical_migration_elecnotes"
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT pg_restore: executing SEQUENCE SET census_historical_migration_elecnotes_id_seq
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:18 RESTORE of table census_historical_migration_elecnotes complete.
   2024-06-04T14:51:18.71-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:18 REMOVING FILE: public-census_historical_migration_elecnotes.dump

_dbrestore: Workflow Run Operations: db to db restore, aws s3 sync

asteel-gsa commented 1 month ago

Reopening as testing has been completed

asteel-gsa commented 2 weeks ago

Once this passes, I am going to merge. We are going to expect that staging and prod fail the scheduled backups, as code will not be on the instance yet

github-actions[bot] commented 2 weeks ago

Terraform plan for dev

Plan: 2 to add, 0 to change, 0 to destroy.

```diff Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # module.dev-backups-bucket.cloudfoundry_service_instance.bucket will be created + resource "cloudfoundry_service_instance" "bucket" { + id = (known after apply) + name = "backups" + replace_on_params_change = false + replace_on_service_plan_change = false + service_plan = "021bb2a3-7e11-4fc2-b06b-d9f5938cd806" + space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" + tags = [ + "s3", ] } # module.dev.module.snapshot-database.cloudfoundry_service_instance.rds will be created + resource "cloudfoundry_service_instance" "rds" { + id = (known after apply) + json_params = jsonencode( { + storage = 50 } ) + name = "fac-snapshot-db" + replace_on_params_change = false + replace_on_service_plan_change = false + service_plan = "815c6069-289a-4444-ba99-40f0fa03a8f5" + space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" + tags = [ + "rds", ] } Plan: 2 to add, 0 to change, 0 to destroy. Warning: Argument is deprecated with module.dev-backups-bucket.cloudfoundry_service_instance.bucket, on /tmp/terraform-data-dir/modules/dev-backups-bucket/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket": 14: recursive_delete = var.recursive_delete Since CF API v3, recursive delete is always done on the cloudcontroller side. This will be removed in future releases (and 6 more similar warnings elsewhere) ```

:x: Error applying plan in Deploy to Development and Management Environment #721

github-actions[bot] commented 2 weeks ago

Terraform plan for meta

Plan: 0 to add, 1 to change, 0 to destroy.

```diff Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: !~ update in-place Terraform will perform the following actions: # module.s3-backups.cloudfoundry_service_instance.bucket will be updated in-place !~ resource "cloudfoundry_service_instance" "bucket" { id = "040c4133-1efe-4281-a485-005960b58405" name = "backups" !~ tags = [ + "s3", ] # (4 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. Warning: Argument is deprecated with module.s3-backups.cloudfoundry_service_instance.bucket, on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket": 14: recursive_delete = var.recursive_delete Since CF API v3, recursive delete is always done on the cloudcontroller side. This will be removed in future releases ```

:x: Error applying plan in Deploy to Development and Management Environment #721

github-actions[bot] commented 2 weeks ago

Terraform plan for dev

Plan: 1 to add, 0 to change, 0 to destroy.

```diff Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # module.dev-backups-bucket.cloudfoundry_service_instance.bucket will be created + resource "cloudfoundry_service_instance" "bucket" { + id = (known after apply) + name = "backups" + replace_on_params_change = false + replace_on_service_plan_change = false + service_plan = "021bb2a3-7e11-4fc2-b06b-d9f5938cd806" + space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" + tags = [ + "s3", ] } Plan: 1 to add, 0 to change, 0 to destroy. Warning: Argument is deprecated with module.dev-backups-bucket.cloudfoundry_service_instance.bucket, on /tmp/terraform-data-dir/modules/dev-backups-bucket/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket": 14: recursive_delete = var.recursive_delete Since CF API v3, recursive delete is always done on the cloudcontroller side. This will be removed in future releases (and 6 more similar warnings elsewhere) ```

:white_check_mark: Plan applied in Deploy to Development and Management Environment #721

github-actions[bot] commented 2 weeks ago

Terraform plan for meta

Plan: 0 to add, 1 to change, 0 to destroy.

```diff Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: !~ update in-place Terraform will perform the following actions: # module.s3-backups.cloudfoundry_service_instance.bucket will be updated in-place !~ resource "cloudfoundry_service_instance" "bucket" { id = "040c4133-1efe-4281-a485-005960b58405" name = "backups" !~ tags = [ + "s3", ] # (4 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. Warning: Argument is deprecated with module.s3-backups.cloudfoundry_service_instance.bucket, on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket": 14: recursive_delete = var.recursive_delete Since CF API v3, recursive delete is always done on the cloudcontroller side. This will be removed in future releases ```

:white_check_mark: Plan applied in Deploy to Development and Management Environment #721

GSA-TTS / FAC