search

praekeltfoundation / marathon-acme

Automate ACME certificates for Marathon apps served by marathon-lb
MIT License
8 stars 3 forks source link

Support for scaling Marathon-LB #138

Closed ti-mo closed 6 years ago

ti-mo commented 6 years ago

Hi, I've been wondering for a while if it would be possible for marathon-acme to support notifying all members of a multi-instance scaled marathon-lb app. Looking around the source code, I don't seem to find anything that could resolve a hostname to multiple A records. As far as I can see, only one instance is notified of a new certificate, the others will happily keep serving the hostname with the default certificate.

eg.:

dig marathon-lb.marathon.mesos
...
;; ANSWER SECTION:
marathon-lb.marathon.mesos. 60  IN      A       10.1.2.3
marathon-lb.marathon.mesos. 60  IN      A       10.1.2.4
marathon-lb.marathon.mesos. 60  IN      A       10.1.2.5

Is this something marathon-acme could support?

Thanks!

jerith commented 6 years ago

I'm sure it is, but I don't think we'll have the capacity to implement it ourselves anytime soon. If you'd like to do the heavy lifting (such as it is) and submit a pull request, we'll be happy to review and help out.

As a workaround, both the --lb and --marathon command line options already accept a comma-separated list of addresses (for example, --lb http://lb1.example.com:9090,http://10.1.2.3:9090) which is how we currently notify all of our marathon-lb instances. It's a little less convenient than using DNS with multiple A records, but for load balancers (which tend not to move around much) the static configuration hasn't been an issue for us.

ti-mo commented 6 years ago

@jerith Thanks for the reply, will try to wrap my head around Twisted. :slightly_smiling_face:

The issue with single-app scaling is the fact that there is only one deterministic hostname for the service. The instances of /marathon-lb can only be discovered by querying A (or the SRV equivalent of) marathon-lb.marathon.mesos. The Mesos tasks created by this app are not independently identifiable by hostname.

jerith commented 6 years ago

Can you use IPs instead of hostnames? (Not suggesting this as a long-term solution, but it should do the job while we figure out something better.)

ti-mo commented 6 years ago

Hmm. Great suggestion, this is one of the very few services we have that use host networking! We can set up marathon-acme to send a hup signal to all public slaves, regardless of whether or not marathon-lb is running on it. Thanks!

ti-mo commented 6 years ago

By the way, this is working for us, feel free to re-open if you want to track this further @jerith. Thanks a lot for the help!