Bulletproof production deployment

[007]

This started as a comment on #379, but I think it should be a separate issue for commentary / resolution. #59 might address some of this process as well, and I'll be using that as my primary reference.

It's great that this has a "do-everything-for-you" script to get started, but a production instance probably shouldn't be using that - it ought to have a list of dependencies, and expect a sysadmin to deploy those things separately from this code. There is no separation of concerns in the current process, and it's unclear whether or not I could run two instances in a HA or load-balanced configuration.

I'm currently working through the steps to set this up as a "hosted" install on AWS, including separate servers / services for each of the required components. I'll document the process and complications as I go, but what's the best way to get that info back upstream?

[timabbott]

So I recently did some refactoring of the puppet modules to make them more modular; you can see the list of modules making up the all-in-one system in puppet/zulip/manifests/voyager.pp.

What I'd love to see happen on the documentation front is a two-step process:

(1) Patch scripts/lib/install script to support taking the deployment type as an argument and thus be able to setup a server for any of the roles making up a Zulip production installation. (You may want to do this on top of the work to the install script in #450 to avoid merge conflicts). You may want to read tools/deprecated/install-server (which is what we used to use to setup new boxes at Zulip Inc) as well.

(2) With that done, the steps to document become a lot simpler, and I think could just be short little guides in README.prod.md for "here's what you do if you want rabbitmq to be its own machine"

I think with that, it'd be a lot easier to set things up in a good way. I should also note that puppet/zulip_internal/ has some config from zulip.com that may be helpful for understanding how to structure a system with separate servers for each of the components.

On the front of high-availability, there's currently no support for a load-balanced configuration but README.prod.md, https://github.com/zulip/zulip/issues/54, #395, #445, and #453 all have relevant content.

[VincentSC]

I agree with Bond, that the scripts don't give the control an administrator needs. As the documentation is not clear what is recommended and required, it neither is clear what is being done by the scripts.

My suggestion: split the documentation for the production into two parts: scripted for Ubuntu 14.04 and manual for the rest. Now more information about dependencies is found in the dev-documentation.

I'm happy to help you write the documentation, once it actually works on anything else than 14.04.

[brainwane]

@christi3k since you're reworking a lot of installation-related documentation, I wanted to give you a heads-up about this suggestion:

split the documentation for the production into two parts: scripted for Ubuntu 14.04 and manual for the rest.

@VincentSC Incidentally, there's fairly new "Installing manually on UNIX-based platforms" documentation, recently polished by @christi3k, which is for manual installation of developer instances (so, not including a bunch of components that production installations need). I presume this is the sort of thing you hope to see for production installation guidance as well.

[timabbott]

Just to follow up on an item in this thread, we've supported 16.04 for some time, and adding Debian I expect would be pretty straightforward.

[brainwane]

One more potential item for this: checking the SHA-1 or MD5 of the https://www.zulip.org/dist/releases/ package and verifying it.

[rht]

What about SHA-256 instead? E.g. https://bitcoin.org/bin/bitcoin-core-0.15.0.1/

[timabbott]

Closing this, since I think we've done most of the generic work that can be done on making Zulip more portable. Remaining work would be for specific platforms we want to support.