GPL-643 UAT PSD LIMS infrastructure from eta to theta

[rl15]

User story GPL-643 | Migrate UAT PSD LIMS infrastructure from eta to theta

Who are the primary contacts for this story Rich L

Acceptance criteria To be considered successful the solution must allow:

• [ ] 17 UAT instances (in smrtlink)

Additional context " We aim to have 50% of machines to be migrated to the new platform during September, and the migration to be complete by the end of October " Suggest UAT (~1/3 of our infrastructure) needs to be moved by ~25th Sept

https://docs.google.com/document/d/1WHyTRtaswzwwoGr1r1ds_fBd1ewOBjSX3FzIB7f4mUI/edit?usp=sharing

[harrietc52]

psd-provison

• [x] create new workspace
• [x] create floating ips
• [x] key pairs in vars and in openstack
• [x] run packer to create images - run docker_based_image with ss_test_vars.
• [x] SMRTlink
• [x] update variables.tf file (uat-theta-....): auth, tenant name, images, floating ips, auth url
• [x] run psd-provision for all instances

psd-config

• [x] add a new environment for uat-theta
• [x] add key to ssh repo
• [x] run dns a name, e.g. theta-uat-traction -> floating ip
• [x] run config_all for all hosts
• [x] run config_docker_hosts for all docker hosts
• [x] run psd_config
• [x] run SMRTlink
• [x] run application specific playbooks
• [x] copy over any volumes - (lighthouse reports, crawler backups, kibana logs)

psd-deployment

• [x] set up scripts
• [x] add environment variables
• [x] check vars for each role
• [x] deploy playbook for each application
• [x] check everything
• [x] cnames
• [ ] Systems certificate update domain ?

post psd-deployment

• [x] Anames eta: Weve updated -uat to .uat-eta in psd-config And we have updated the eta vars in psd-deployment, but havent ran any deployment So currently eta is unaccessible, but ready to be deployed if necessary Redeployed labwhere to training on eta as it points to uat-theta rmq theta: We've kept the a names the same And we updated some vars in psd-deployment to make sure hard coded a names for theta are correct

• [x] Stop UAT eta
• [ ] Ask everyone to test

[harrietc52]

Unexpected:

• Had to increase flavour on consul machine as ran out of disk space
• Monit increase in size
• Couldnt back up mmonit database as it was sqlite and corrupted. Change to mysql
• remember to reload nginx

[harrietc52]

Future TODO

• Speak to Systems about generating/ updating certificate?
• Is netdata used?
• Old uat servers are down. Get snapshot of each machine in eta, and upload snapshot in theta. So keep backup in theta. Openstack commands.
• Create new database in mysql from systems. MMonit database using sqlite, will break eventually. Need to move to mysql/ postsql in uat theta. Replicate database in mysql. Change config, restart monit.
• Monit license check if it going out of data.
• Move PMB into own stack and out of samples extraction stack. Terraform/config/deployment/little docker. New manager machine. Downgrade saex to worker. create pmb worker. cluster of three nodes. diagram here
• Anames currently have both -uat and -uat-theta. Clean up and remove *-uat-theta. This will involve updating psd-deployment and UAT downtime

[harrietc52]

Cnames

1. git mv uat folder in enviroment to to uat-eta
2. git mv uat-theta folder in environmnent to uat
3. remove the cnames for uat-theta. this will involve running the current uat enviroment with state absent
4. update cname vars for enviroment uat. which involves renames uat-theta. to uat.
5. drop cnames for uat-eta
6. create cnames for uat
7. Redeploy all apps

[harrietc52]

Tomorrow: merge PRs ask everyone to test