DMPHub prod deploy

[briri]

Build out the DMPHub prod environment and then connect DMPTool prod to it

December:

• [x] Write up task list and plan
• [x] Setup the base SSM parameters (outlined in the wiki docs)
• [x] Run Sceptre builds for: sqs.yaml (includes s3.yaml), eventbridge.yaml, dynamo.yaml
• [x] Build HostedZone for prod: 'dmphub.uc3prd.cdlib.net'. See issue #76
• [x] Finish incorporating new homepage into DMPTool (deploy to stage)
• [x] Run Sceptre builds (once HostedZone is in place) for: cognito.yaml (includes route53.yaml, cloudfront.yaml, cert.yaml, waf.yaml), eventbridge.yaml and then run SAM build/deploy
• [x] Deploy the React Landing Page app cd src/landing_page && ruby build_deploy.rb prd
• [x] Prep DMPTool codebase (once HostedZone is in place)
  • [x] Update URLs for new DMPHub Endpoints on local machine
  • [x] Add mechanism to restrict React pages to certain Orgs
  • [x] Discuss with @mariapraetzellis how best to provide access to the new React Prototype. Should we add it to the Nav menu as ‘Upload’?
• [x] Ensure EZID minting of new DOIs is disabled aws dynamodb put-item --table-name [Table ID] --item "{\"PK\":{\"S\":\"PROVENANCE#$KEY\"},\"SK\":{\"S\":\"PROFILE\"},\"contact\":{\"M\":{\"email\":{\"S\":\"$ADMIN_EMAIL\"},\"name\":{\"S\":\"Administrator\"}}},\"description\":{\"S\":\"The $2 $1 system\"},\"downloadUri\":{\"S\":\"https://$3/api/v2/plans/\"},\"homepage\":{\"S\":\"https://$3\"},\"name\":{\"S\":\"$2\"},\"redirectUri\":{\"S\":\"https://$3/callback\"},\"tokenUri\":{\"S\":\"https://$3/oauth/token\"},\"seedingWithLiveDmpIds\":{\"BOOL\":true}}" and set the SSM parameter: aws ssm put-parameter --name /uc3/dmp/hub/prd/EzidPaused --overwrite --value true --type String
• [x] Test local DMPTool against new Prod API endpoints and landing page by: registering a DMP, then make sure updates and narrative are working
• [x] Ensure that the Admin email setup in AWS receives an error message from the EZID Publisher Lambda that indicates that the DMP ID needed to update EZID, but that EZID is paused
• [x] Verify landing pages are working properly

Week Prior to Deploy:

• [x] Delete the test record from December from the Dynamo table (pending permission update) For example: aws dynamodb delete-item --table-name my_table --key '{"PK":{"S":"DMP#doi.org/10.48321/D143E22b00"},"SK":{"S":"VERSION#2023-12-19T16:54:29Z"}}' && aws dynamodb delete-item --table-name my_table --key '{"PK":"DMP#doi.org/10.48321/D143E22b00","SK":"VERSION#latest"}' && aws dynamodb delete-item --table-name my_table --key '{"PK":"DMP#doi.org/10.48321/D1262B1640","SK":"VERSION#latest"}' && aws dynamodb delete-item --table-name my_table --key '{"PK":"DMP#doi.org/10.48321/D16426ebee","SK":"VERSION#latest"}' && aws dynamodb delete-item --table-name my_table --key '{"PK":"DMP#doi.org/10.48321/D10F13a9e1","SK":"VERSION#latest"}'
• [ ] Set DMPTool maintenance message
• [x] Test migration of a few existing DMPs by:
  • [x] Run rails db:migrate -e production
  • [x] Manually set the new plans.dmp_id field for the DMPs we want to test with (maybe just UCOP owned ones like FAIR ISLAND?).
  • [x] On local machine pointed at prod DB and new DMPHub prd, run rails v5:seed_dmps to ensure that the DMP ID metadata moves over correctly and that the PDF narratives are also moved
• [x] Enable EZID aws dynamodb put-item --table-name [Table ID] --item "{\"PK\":{\"S\":\"PROVENANCE#$KEY\"},\"SK\":{\"S\":\"PROFILE\"},\"contact\":{\"M\":{\"email\":{\"S\":\"$ADMIN_EMAIL\"},\"name\":{\"S\":\"Administrator\"}}},\"description\":{\"S\":\"The $2 $1 system\"},\"downloadUri\":{\"S\":\"https://$3/api/v2/plans/\"},\"homepage\":{\"S\":\"https://$3\"},\"name\":{\"S\":\"$2\"},\"redirectUri\":{\"S\":\"https://$3/callback\"},\"tokenUri\":{\"S\":\"https://$3/oauth/token\"},\"seedingWithLiveDmpIds\":{\"BOOL\":false}}", set the SSM parameter: aws ssm put-parameter --name /uc3/dmp/hub/prd/EzidPaused --overwrite --value false --type String and test single register/update and verify EZID updates. THEN DISABLE AGAIN!
• [x] Run rails v5:upgrade_5_0 -e production to add API info to orgs
• [x] Update dependencies and create release (Set as a PRE-RELEASE since new stuff is not usable by others)

Day before:

• [x] Move all existing DMPs created prior to 2023 in chunks
  • [x] Migrate the DMP IDs to their new field on the plans table - Run rails v5:move_dmp_ids RAILS_ENV=production Update script with .where('created_at <= \'2023-01-01T00:00:00+00:00\'')
  • [x] Transfer the pre-2021 records to the new DMPHub - Run rails v5:seed_dmphub RAILS_ENV=production Update script with .where('created_at <= \'2021-01-01T00:00:00+00:00\'')
  • [x] Transfer the pre-2022 records to the new DMPHub - Run rails v5:seed_dmphub RAILS_ENV=production Update script with .where('created_at <= \'2022-01-01T00:00:00+00:00\'')
  • [x] Transfer the pre-2022 records to the new DMPHub - Run rails v5:seed_dmphub RAILS_ENV=production Update script with .where('created_at <= \'2023-01-01T00:00:00+00:00\'')
  • [x] Transfer the pre-September 2023 records to the new DMPHub - Run rails v5:seed_dmphub RAILS_ENV=production Update script with .where('created_at <= \'2024-01-01T00:00:00+00:00\'')
  • [x] Update EZID's target URLs for these newly transferred records - Run rails v5:update_ezid_targets RAILS_ENV=production Update script with .where('created_at >= \'2024-01-01T00:00:00+00:00\'')

Deploy:

• [x] Create release v5.2
• [x] Disable DMP registration in DMPTool @briri
• [x] Deploy the v5.2 tag @briri
• [x] Install delayed_job on host @ashleygould
• [x] Ensure URLs for new DMPHub Endpoints are in place (credentials file) aws ssm put-parameter --name credentials_yml_enc --type SecureString --overwrite --value [creds file].
• [x] Run bin/rails config:trace to verify DMPHub credentials are correct. @briri
• [x] Run query to detect any DMPs that may have been already been migrated but have been updated since then. select id from plans where dmp_id IS NOT NULL AND update_at > '2024-01-18T14:00:00+00:00'
• [x] Run rails v5:move_dmp_ids -e production to move DMP ids to plans.dmp_id from identifiers Update script with .where('created_at >= \'2023-09-01T00:00:00+00:00\'')
• [x] Run rails v5:accept_terms_plans -e production to accept the terms for all plans that are already publicly visible
• [x] Ensure DMPTool profile in DMPHub is set to 'seed' so we do not register new DOIs via EZID aws dynamodb put-item --table-name [Table ID] --item "{\"PK\":{\"S\":\"PROVENANCE#$KEY\"},\"SK\":{\"S\":\"PROFILE\"},\"contact\":{\"M\":{\"email\":{\"S\":\"$ADMIN_EMAIL\"},\"name\":{\"S\":\"Administrator\"}}},\"description\":{\"S\":\"The $2 $1 system\"},\"downloadUri\":{\"S\":\"https://$3/api/v2/plans/\"},\"homepage\":{\"S\":\"https://$3\"},\"name\":{\"S\":\"$2\"},\"redirectUri\":{\"S\":\"https://$3/callback\"},\"tokenUri\":{\"S\":\"https://$3/oauth/token\"},\"seedingWithLiveDmpIds\":{\"BOOL\":true}}" and set SSM parameter: aws ssm put-parameter --name /uc3/dmp/hub/prd/EzidPaused --overwrite --value true --type String
• [x] Run rails v5:seed_dmphub -e production to write DMPs to the new DMPHub
• [x] Update EZID's target URLs for these newly transferred records - Run rails v5:update_ezid_targets RAILS_ENV=production
• [x] Run rails v5:build_narratives -e production to build public plan PDFs and store in S3
• [x] Remove 'seed' flag from DMPTool profile in DMPHub so that new registrations and updates will write to EZID aws dynamodb put-item --table-name [Table ID] --item "{\"PK\":{\"S\":\"PROVENANCE#$KEY\"},\"SK\":{\"S\":\"PROFILE\"},\"contact\":{\"M\":{\"email\":{\"S\":\"$ADMIN_EMAIL\"},\"name\":{\"S\":\"Administrator\"}}},\"description\":{\"S\":\"The $2 $1 system\"},\"downloadUri\":{\"S\":\"https://$3/api/v2/plans/\"},\"homepage\":{\"S\":\"https://$3\"},\"name\":{\"S\":\"$2\"},\"redirectUri\":{\"S\":\"https://$3/callback\"},\"tokenUri\":{\"S\":\"https://$3/oauth/token\"}}" and set SSM parameter: aws ssm put-parameter --name /uc3/dmp/hub/prd/EzidPaused --overwrite --value false --type String
• [x] Run the seed_dmphub script again but for the Plans that have been updated over the last week (see step 5)
• [x] Enable DMP registration in DMPTool
• [x] Test by updating test DMP (updates new Hub and narrative as well as EZID record)
• [x] Test DMPUpload functionality
• [ ] Run rails v5:enable_v5_pilot to set the v5_pilot flag for pilot partner orgs so that they can access the new React pages (make sure all org ids are in the Array first!)
• [ ] Run a query to purge any DMP IDs from the old identifiers table that have been migrated: delete from identifiers where identifier_scheme_id = 8 and identifiable_type = 'Plan' and value like 'https://doi.org/10.48321/%' and identifiable_id in (select id from plans where dmp_id is not null and dmp_id != '');
• [] Search for any remaining DMP IDs in the identifiers table and reconcile select * from identifiers where identifiable_type = 'Plan' AND value like 'https://doi.org/10.48321/%';

Post Deploy:

• [ ] Determine appropriate time to decommission DMPHub EC2 instances in the old main AWS account

[briri]

@ashleygould, can you please give me permission to execute dynamodb:DeleteItem in the stg/prd account?

[briri]

delayed job is having an issue starting up via systemd but we were able to get it running when starting it manually. Need to investigate tomorrow

Forgot to replace the credentials file BEFORE deploying which caused issues when trying to contact the DMPHub. Had to update the SSM parameter and then redeploy the app

[briri]

DMP Upload seems to be working except for the NIH project search. Which records the following in the logs. Interestingly, it is showing localhost so something appears to be wrong in the React client.

I, [2024-01-23T19:34:06.736010 #8572]  INFO -- : [c4166358-e0c9-427b-9540-2f1df85775b6] Started GET "/localhost/api/v3/awards/nih?title=&years=2023&pi_names=Smith" for 128.48.94.168 at 2024-01-23 19:34:06 -0800
F, [2024-01-23T19:34:06.736801 #8572] FATAL -- : [c4166358-e0c9-427b-9540-2f1df85775b6]   
[c4166358-e0c9-427b-9540-2f1df85775b6] ActionController::RoutingError (No route matches [GET] "/localhost/api/v3/awards/nih"):

[briri]

Was able to fix the funder project search. Issue was with script that populated DB records

[briri]

This is complete. Will create a new ticket for the delayed_job issue