Annual Operational Tasks - including credential rotation

[terrywbrady]

Now that we have retired the old mrt-conf-prv repo, I think it is time to tackle credential rotations. I was thinking about creating an annual calendar of big operational tasks we should perform. Perhaps we could calendarize these by month or by sprint.

Here are some ideas that I have for what we might want to calendarize.

• Marisa or Eric will calendarize

• Rotate RDS credentials - 2x/yr

  • Document proposed process (toggle invrw, invro, billingro, billingrw)
  • One time manual: create new users invrw2, invro2, billingro2, billingrw2
    • find way to script permission "grant" statements for new users
    • or create user groups for this purpose
    • create dedicated invro user for the etd process - not SSM configured
  • create script to set SSM vars
    • generate password
    • set database password (need to be very careful about managing the credentials that can perform this task)
    • update SSM (user and password)
    • confirm sql login
  • restart ALL services
    • pause dryad ingests (because of sword)
    • pause large object assemblies
    • pause merritt ingests
    • use patching cycle logic to systematically take servers out of load balancers and perform restart
  • Check for any stray db connections using the DBA account
  • Automated testing
    • run integration test suite
    • run consistency reports
  • Reset passwords for prior user ids
    • Verify: prior versions are still accessible via SSM calls in case fallback is needed
    • Push random password change. Password will not be logged anywhere. Only the active SSM vals will be used
  • Run automated testing again to confirm that no processes still depended on the old values
  • Cycle the credentials for the DBA account and update SSM
  • Alternatives to this approach
  • Perform at least twice a year and when team members depart
  • Consider Secrets Manager (service code changes required)
  • Consider encrypting SSM (service code changes required)

• Rotate Storage credentials - 2x/yr

  • David will investigate if the toggling credentials is possible with SDSC and Wasabi
  • Implement after RDS changes are made
  • Remember to separate stage from prod

• Storage scans - 3x/yr

• Send invoices - 1x/yr

• Update storage estimates - 2x/yr

• AWS billing and Librato review - ?x/yr

  • Review Librato graphs at each Sprint retro (look at past 3-4 weeks)
  • RDS
  • Store
  • Ingest
  • Audit
  • UI
  • OAI
  • Sword
  • Access
  • Inventory
  • S3
  • ZFS

• Repository and Repo tagger config review - 1x/yr

• Review SSM registry

[marisastrong]

@marisastrong and @elopatin-uc3 look at calendar options - putting recurring events on the UC3 calendar. can invite others as well to event reminder

[elopatin-uc3]

@marisastrong @terrywbrady @elopatin-uc3 discussed adding a meeting with Finance to the list of annual tasks. We will schedule this for the first week of December.

[terrywbrady]

@elopatin-uc3 , @dloy , @ashleygould conducted the stage rotation on 12/8/22.

Based on the fact that ingest/dryad will be paused, there is no need to remove resources from the load balancer.

The following services will require a restart:

• ui
• inv
• audit
• replic
• oai
• sword