Mup 1.5 Plan - Load balancing and zero downtime deploys

[zodern]

Mup 1.5 will add support for load balancing and zero downtime deploys. It will use Docker Swarm and the reverse proxy to implement this. Mup 1.5 does add an experimental swarm integration, but it is not required for load balancing or zero downtime deploys.

• [ ] Docker plugin
  • [x] Setup docker swarm. Done in mup 1.4
  • [x] Show swarm status in mup docker status. Done in mup 1.4
  • [ ] Discover and fix edge cases
  • [x] Document restrictions. In this initial release, mup will only support one app using swarm per server. Other apps can be deployed to the same servers as long as they do not use swarm. Also, it will only support deploying apps when Prepare Bundle is enabled due to how services rollback versions. It will be recommended to use servers with at least 1gb ram with Docker Swarm.
  • [ ] Recover when quorum is lost
  • [x] Show error when servers in config are part of several clusters
  • [x] Show error when servers are part of a cluster, but mup doesn't have access to a manager in that cluster
  • [x] Decide how to handle nodes and managers that are not in the list of servers in the config
  • [x] Command to destroy the swarm cluster. Done in mup 1.4
• [ ] Meteor plugin
  • [x] Use specific numbers for an app's image tag instead of previous and current to support how services roll back failed deployments
  • [x] Deploy app as a swarm service
  • [ ] Explore possibilities of validating deployment
• [ ] Plugin API
  • [x] Review swarm api's that have already been added
  • [ ] Document api's
  • [ ] ~Give user more control if desired over what servers are managers~
  • [x] Add general tasks any plugin can use:
  • [x] Create service
  • [x] Update service
  • [x] createOrUpdateService
  • [ ] Wait until service is ready
    • [ ] Decide how to handle a service that encounters an error starting since it repeatedly retries
    • [ ] Allow plugin or Meteor Up to look at output to offer suggestions for fixing errors
• [ ] Mongo (optional)
  • [ ] Deploy mongo as a service with an overlay network to support the app running on multiple servers
• [ ] Proxy
  • [ ] ~Add support to nginx-proxy for docker swarm, or run docker flow, dockercloud haproxy, or traefik between nginx-proxy and the app~
  • [x] Run container on proxy's servers for each service using the proxy (needed for the reverse proxy to know about the service)
  • [x] Create container to generate upstream blocks for services
  • [x] Create fork of nginx-proxy that supports custom upstream blocks
  • [x] Connect apps to same overlay network as proxy
  • [x] Only run the proxy on manager nodes
  • [ ] ~Show what servers to use in the dns records~
  • [ ] Graceful shutdown
• [ ] Default docker image
  • [ ] Check compatibility between meteorhacks:meteord and zodern/meteor
  • [ ] Consider changing the default docker image to zodern/meteor

[stefanve]

Is there any timeline for this?

[zodern]

I first would like to finish more of the tasks in the 1.4 project. I might also try to reduce problems with ECONNRESET. After that I will start working on 1.5.

Anyone is welcome to start working on these changes. If you have questions you can ask on gitter or create a new issue.

[pagesrichie]

Just wanted to clarify - I was looking through the meteor up documentation and noticed there were options in the json file for multiple servers, so I automatically assumed load balancing would come with it. So that option is there for uploading to them one by one, however load balancing is not a part of mup yet and is in the pipeline for mup 1.5, correct?

[zodern]

That is correct.

[pagesrichie]

Is this a good article to resolve the load balancing issue: http://joshowens.me/building-your-own-meteor-galaxy-hosting-setup/ , or setting up load balancing droplet on Digital Ocean as well?

[zodern]

Either option would work.

[pagesrichie]

Thanks. Looking forward to your solution for mup 1.5! When do you expect it to be finished (load balancing part especially)?

On Sat, Apr 7, 2018 at 11:06 AM, zodern notifications@github.com wrote:

Either options would work.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/zodern/meteor-up/issues/893#issuecomment-379488258, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab4qvM6T6MWqYkXpLsRXFrPvGFOnsbucks5tmQAbgaJpZM4SPorK .

-- Richie Pages Web developer/founder/writer

[LazerJesus]

any progress toward zero downtime deploys?

[zodern]

I have done some work the past couple months on this, mostly investigating the different options for using Docker swarm with the nginx-proxy container and starting work on a solution for it. I will be working part time on Mup next month.

[LazerJesus]

thank you. thats a really great feature!

[zodern]

A beta for mup 1.5 is now available. Learn more in #976.

[WayneUong]

@zodern you meant https://github.com/zodern/meteor-up/pull/975 ?

[zodern]

Thanks @WayneUong. It's now #976 since I had used the wrong branch in the pull request.

[CJjumper]

Hi @zodern congrats on the amazing work you have been doing, can we have an update on this list? What the things have changed since April? Would be amazing to track your progress as you go.

[zodern]

The second beta is now available. Most of the changes have been around bug fixes and adding missing aspects of swarm support, and general improvements (many implemented by @rolljee). Zero downtime deploys mostly works now. I've updated https://github.com/zodern/meteor-up/pull/976#issue-200468540 with the current status.

What is left: https://github.com/zodern/meteor-up/projects/4 The biggest tasks remaining are informing the user and show logs from failed tasks when a deploy fails, and improving rolling deploys to support apps that are slow to start.

[rolljee]

Looking forward to test that !

[zodern]

Mup 1.5 is almost ready. It is fairly stable, and I am using it in production for a couple of projects. I've updated the status at https://github.com/zodern/meteor-up/pull/976. There have been a few new improvements to performance (an new option that makes Prepare Bundle faster, and using a private docker registry for faster deploys to multiple servers), and a simpler load balancing implementation that doesn't use Docker Swarm.

Swarm is almost complete, but there are two issues left. I am not sure if they will be fixed before Mup 1.5 is released:

• It does not clean up old images, so after enough deploys the server will run out of disk space
• When a deploy fails, it does not show the logs. The docker cli doesn't either when updating a service, and implementing it could be a challenge.

[pagesrichie]

Looking forward to this!! Thanks for the update!

On Mon, Feb 24, 2020 at 10:42 PM zodern notifications@github.com wrote:

Mup 1.5 is almost ready. It is fairly stable, and I am using it in production for a couple of projects. I've updated the status at #976 https://github.com/zodern/meteor-up/pull/976. There have been a few new improvements to performance (an new option that makes Prepare Bundle faster, and using a private docker registry for faster deploys to multiple servers), and a simpler load balancing implementation that doesn't use Docker Swarm.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/zodern/meteor-up/issues/893?email_source=notifications&email_token=AG7CVPG47QNKG22LYFOGLIDRESHUBA5CNFSM4ER6RLFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM2OBMA#issuecomment-590667952, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG7CVPBW6WGQ5323ZIXQN3DRESHUBANCNFSM4ER6RLFA .

-- Richie Pages Web developer/founder/writer

[zodern]

Beta 8 is now available. I did thorough testing of load balancing and running multiple apps on a server to make sure there is no unnecessary downtime, and beta 8 has all of the issues I found fixed.

[zodern]

Beta 9 was published today to add debugging of apps in production. Run mup meteor debug, and you can use your local node debugging tools to debug your app in production.

[zodern]

Beta 10 was published today to fix mup meteor debug. Also, all new situations since 1.4.x where mup does not properly clean up disk usage should be fixed, except for the swarm integration.

[longyarnz]

I don't know if this is the place to state this but mup 1.5 breaks mongo authentication. I tried migrating to 1.5 from 1.4.6. I get a mongo authentication error while running 1.5. I downgraded and the error vanished. I upgraded again and the error returned. I use an external mongo server with a replica set.

[zodern]

Beta 14 is now available, with all new bugs that I know of fixed. This should be the last beta. I plan to release the stable version next week.

Try it with npm i -g mup@next.

[pisacode]

@zodern how do should we set the domain setting for load balancing, right now we use digital ocean load balancer to point the domain to multiple droplet ips and also we set the proxy.loadBalancing true . However the docs never mentions setting up a digital ocean load balancer or any sort of that?

If setting proxy.loadBalancing true is enough how can we point the domain to two different servers?

Thank you for the amazing package.

[zodern]

Hi @pisacode. if you are using a Digital Ocean load balancer, you probably don't need to use Meteor Up's load balancer. It might be helpful to better understand what you are trying to do, or why you want to use both load balancers (I know some companies do for various reasons).