[RELEASE] Ingest & Inventory - new queue library

[mreyescdl]

Overview of Release

Large set of changes involving new queue structure and many more Ingest states /ingest & /inventory queue obsolete

Service(s)

Ingest & Inventory

Github Issue Link(s)

Too many to list

Servers affected

uc3-mrtingest-prd01 uc3-mrtingest-prd02 uc3-mrtingest-prd03 uc3-mrtinv-prd01 uc3-mrtinv-prd02

Source Code:

Ingest - https://github.com/CDLUC3/mrt-ingest/pull/105 Inventory - https://github.com/CDLUC3/mrt-inventory/pull/33

Source Code Tag

Ingest - 2.1.1 Inventory - 2.1.1

Jenkins/Build Info: (if applicable)

N/A - Using Code Artifact

Deployment Steps (if applicable)

• [x] 10/7, 11AM - Pause Ingest
• [x] 12:30PM - Modify Hiera data for Ingest production hosts (new module, additional ENV vars)
• [x] Redeploy Inventory workers to tag 2.1.1
• [x] Merge "production-queue" profile branch into "production"
  • Pull request: https://github.com/cdlib/mrt-ingest-profile/pull/129
• [x] Apply profiles into ZFS
• [x] Redeploy Ingest workers to tag 2.1.1
• [x] Convert old queue data to new queue (Terry's script)
  • create ZK node /migration/m1 to indicate that admin tool should read from the new queues
• [x] Submit test object to new Ingest worker
• [x] Thaw Ingest

Infrastructure Changes to support deployment (ALB, EFS, etc)

None needed - Ansible script will perform deployment for Ingest and Inventory

Rollback Instructions

• [ ] Use archived WAR file for Ingest and Inventory for redeployment
• [ ] Ingest - 1.0.26
• [ ] Inventory - 1.8.3
• [ ] Remove ZK node /migration/m1
• [ ] Copy archived profiles into place (/dpr2/ingest_home/profiles.20241003)
  • Profile rollback can be done by reverting to tag "rollback_preZK"
• [ ] Turn off profile cron

[mreyescdl]

Found bug after production deployment dealing with updates vs adds. https://github.com/CDLUC3/mrt-ingest/commit/b2db03b2603e93f863ed4e3c00764b5ccd0d8531

Decided to not rollback. Paused Ingest queue and applied hot-fix. Released queue and monitored.