search

mozilla / identity-ops

Tools and Chef cookbooks used by Mozilla Services Operations to provision and manage Persona
Other
24 stars 12 forks source link

Deploy current train-2014.04.16 to prod :beers: #147

Closed jaredhirsch closed 10 years ago

jaredhirsch commented 10 years ago
gene1wood commented 10 years ago

0523 stacks created in production in both regions with the ELBs

persona-org-0523-1203276842.us-east-1.elb.amazonaws.com yahoo-login-persona-org-0523-410505119.us-east-1.elb.amazonaws.com gmail-login-persona-org-0523-839261773.us-east-1.elb.amazonaws.com internal-keysign-0523-1193424009.us-east-1.elb.amazonaws.com internal-dbwrite-0523-1006273021.us-east-1.elb.amazonaws.com

persona-org-0523-972341132.us-west-2.elb.amazonaws.com yahoo-login-persona-org-0523-448428520.us-west-2.elb.amazonaws.com gmail-login-persona-org-0523-2056340272.us-west-2.elb.amazonaws.com internal-keysign-0523-559614322.us-west-2.elb.amazonaws.com internal-dbwrite-0523-217965495.us-west-2.elb.amazonaws.com

gene1wood commented 10 years ago

I set the new 0523 stacks like at 2:55pm PDT. We tested and found an issue with air.mozilla.org and mozillians.org. I reverted the deploy at 3:12pm PDT. I also fumbled the identity-bridge changes resulting in a 1 to 2 minute outage for @gmail.com and @yahoo.com users.

We're now back on the 0324 stacks

gene1wood commented 10 years ago

Root cause appears to be mozilla/persona#4131 once we dig into that we can try another deploy