establish EIPs for mozidp prod servers

[gene1wood]

Currently the production mozidp servers do not have elastic IPs associated with them. Consequently when the instances restart they get different dynamic public IP addresses.

This results in a few things

• monitoring is mapped to IP so our monitors become detached from the instance
• DNS is statically mapped to the IP so the host leaves the DNS load balanced group
• DNS healthchecks are statically mapped to the IP so healthchecks now check a different IP than that of the host
• these IPs are whitelisted (I suspect) to the mozilla ldap server and so it loses it's ability to reach ldap

Please

• setup EIPs for the hosts
• update monitoring to use the new EIPs
• update DNS to use the new EIPs
• update the ldap whitelist to use the new EIPs
• update the healthchecks

The event that triggered me noticing this is that the mozidp server in us-west-1 needed to be restarted. I restarted it and it changed it's ip from 54.215.5.141 to 204.236.168.212. I updated DNS and monitoring only to encounter the ldap whitelist issue. I've changed DNS on the host that restarted to login-prod-disabled.allizomaws.com to force it out of the DNS load balance group.

[mostlygeek]

Closing won't fix. There is already a documented procedure in mana about how to deploy mozidp.

[gene1wood]

So what occurred here is that the mozidp servers run in ec2-classic. I was used to VPC based hosts. These 2 environments cause different behavior in regards to EIPs

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#VPC_EIP_EC2_Differences

The result is that the mozidp servers, when they're restarted from the amazon gui (instance start/stop or restart) they lose their binding to their EIP and it must be manually rebound to the host.

I wasn't aware of this difference between ec2-classic and VPC so when I saw the host restart with a new IP I'd assumed (incorrectly) that it hadn't had an EIP. It did have one it was just no longer bound to it.

Benson went in this morning and manually bound the EIP to the instance. He changed DNS back to point to the old IP. I went in and changed monitoring to point to the old IP. And ldap began working again as the whitelist allowed the traffic in on the old IP.