search

eBayClassifiedsGroup / PanteraS

PanteraS - PaaS - Platform as a Service in a box
GNU General Public License v2.0
199 stars 61 forks source link

Multi data center awareness #165

Closed cookandy closed 8 years ago

cookandy commented 8 years ago

Hi,

I was wondering if anyone had any experience running PanteraS over multiple data centers? I know that Consul has support for multiple data centers, but what about Mesos?

How would you configure HA for this solution? I see two options:

  1. Using active/active - where docker containers run on both sides and consul DNS routes the requests to the appropriate containers. I am not sure if this is best practice as the requests might have to traverse public links.
  2. Using active/passive - where Marathon will spin up new instances of docker containers when they fail at the primary site.

I think in order to use the second option, Mesos would need to be configured as a cluster across the 2 data centers.

Does anyone have any experience with this type of configuration?

sielaq commented 8 years ago

I can tell you how we are doing it.

  1. we have full agnostic stack of all components in every DC Running Mesos/Marathon/Consul across multiple DC is not recommended by those apps (only consul are connected with wan option)
  2. DNS do the balance between DC (poor but sufficient)
  3. Every DC have minimum 3 master +n slaves, so If any mesos /marathon /consul (master) dies the other one takeover.

If we have any DC problem we take it out from DNS traffic so this describe your situation 1.

sielaq commented 8 years ago

btw. most problematic for mesos and marathon is zookeeper which is not a best solution for multiple DC.

cookandy commented 8 years ago

Thanks @sielaq. Can you please tell me what the benefit is for running Consul across the WAN? For your DNS balancing between DC, is Consul doing that? I'd imagine you'd need a public DNS balancer (i.e. DynDNS) to handle the inbound requests to public IPs.

Do you use Consul WAN for anything?

sielaq commented 8 years ago

Ad.1 The benefits of using consul across WAN are multiple: a) your services can communicate with other DC in case a local one are unreachable b) specify exactly which service from which DC should be used: your_service.service.dc1.consul For example: On an external PROXY (like Apache ProxyPass / Varnish / Nginx) you can specify from which DC you want to use the service. c) another example little simple one :) you can specify which marathon should be used (from which DC) marathon.service.dc1.consul or marathon.service.dc2.consul

Ad.2 We are not using Consul for external DNS balancing, yes we have a public DNS that is agnostic from PaaS.

cookandy commented 8 years ago

Ok, I understand now. Thanks for the information!