Easy deployment solution

[zorbathut]

It'd be great to turn the deployment system into a one-button deployment, especially for updating. Right now there's no good set of instructions for how to update a running server.

This isn't necessary to roll it live - we can just muddle through if we have to - but it would make things a lot easier.

[Deimos]

These should be most of the steps needed to solely get Tildes to work on a fresh Ubuntu 16.04 server, but doesn't cover all the other things like setting up permissions correctly, ssh security, firewall, etc. Most are the actual commands to run, but comment lines (starting with #) are descriptions of something that needs to be done but not the specific commands to do it.

adduser tildes
usermod -aG sudo tildes

# clone tildes and tildes-static-sites repos into /home/tildes

sudo ln -s /home/tildes/tildes/tildes /opt/tildes

sudo ln -s /home/tildes/tildes/salt/salt /srv/salt
sudo ln -s /home/tildes/tildes/salt/pillar /srv/pillar

sudo ln -s /home/tildes/tildes-static-sites/output /opt/tildes-static-sites

sudo mkdir /etc/salt
sudo cp /home/tildes/tildes/salt/minion /etc/salt/minion

# edit /etc/salt/minion and change id to "prod"

wget -O - https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
echo 'deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial main' | sudo tee /etc/apt/sources.list.d/saltstack.list
sudo apt-get update
sudo apt-get install salt-minion
sudo systemctl stop salt-minion
sudo systemctl disable salt-minion

# copy /opt/tildes/production.ini.example to production.ini and replace placeholders

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx

sudo certbot certonly --manual -d tildes.net -d *.tildes.net -d tild.es -d *.tild.es --preferred-challenges dns-01 --server https://acme-v02.api.letsencrypt.org/directory

sudo salt-call --local state.highstate

As for updating a running server, it should happen automatically on the development version, but for the live server you usually just need to pull the new code from the repo into /opt/tildes (assuming you set it up as above) and then:

1. If there are new Alembic migrations included (changes to database tables), run alembic upgrade head.
2. If there were changes to the Salt states, run sudo salt-call --local state.apply -l info
3. Always run sudo systemctl reload gunicorn.service, which will load the updated code.

[laverya]

To me "how to easily deploy things" is very strongly linked to how this will be run. A CI/CD setup that works great with k8s would be completely unsuitable for a pure VM hosting solution, for instance.

[zorbathut]

I feel like the best answer I have available here is to shrug helplessly and say "I dunno, how do you think this should be run".

That said, my personal current plan is to just shove everything on a reasonably-inexpensive VPS in some way. I'm happy to involve whatever infrastructure makes sense for watchdog restarts and clean updates and so forth, but I don't see much need to go massively enterprise-tier at the moment; the entire site ought to run on a single reasonably inexpensive server.

I would definitely appreciate suggestions because the last time I did anything devops-related was when the first step of every new setup was "install a LAMP stack".

[antonpaquin]

Are you amenable to docker? That's probably the easiest way of turning the infrastructure into code without locking into a cloud provider.

Con: the way this repo is currently set up to run is via saltstack installing and configuring a bunch of services on one machine, which is a different philosophy from the microservice approach.

I'll give it a deeper dive soon-ish (probably tonight)

[laverya]

I think the trouble with deploying via docker would be data persistence - ideally we'd be able to host the database independently (even if it's still on the same server) but I don't know if the app architecture is conducive to that.

[antonpaquin]

As far as I can tell, the only databases involved are postgres and possibly redis. Breaking postgres into its own container should be easy, unless something weird is going on (something weird is always going on). I'll have to dig around to see if redis is a caching thing or if it actually needs to be persistent.

My "deeper dive" is checking to see to what degree I'm able to replace the salt stuff with docker-compose. There's also the question of whether it's worth it to containerize things or better to just stick with a monolithic server.

[Deimos]

Redis definitely needs to be persistent. It's used for:

• All the asynchronous jobs / queues via Redis Streams
• Session storage
• Rate-limiting actions, with a module that implements Generic Cell Rate Algorithm
• Preventing users from using passwords seen in data breaches with a bloom filter (if you load a breached-password dataset into it) - this uses a second Redis server, but you could just drop that one if you don't want this functionality

[zorbathut]

So we ended up batting this around a lot on Discord, and as far as I know, have come to this rough set of conclusions:

• Docker is pretty neat.
• Docker, by default, does a really good job of deployment as long as things are simple. It doesn't handle deployment particularly well if you have complicated needs and it doesn't do a good job of updating.
• Docker with an orchestrator handles all of that stuff extremely well . . .
• . . . but there aren't any good Docker orchestrators designed for small-scale sites.
• Given the the thing we're looking for is somewhat-complicated deployment and updating on a small-scale site, Docker may just not suit our needs right now . . .
• . . . doubly so because Tildes isn't currently built on Docker.
• This all seems like potentially a lot of fuel for some neat long-term projects, including someone writing a small-scale orchestrator designed for one-to-ten-computer clusters, and if you want to tackle that, go for it, but it is absolutely not a thing I'm worrying about right now.

Given that Tildes is currently deployed and updated with shell commands, I think the right solution here is just to write some simple and reasonably-tested shell or python scripts to do the deploy and update processes. With luck we can build an update script that'll suit @Deimos 's needs at the same time.

(As an example, one thing that scares me about what you've described there is that you update the database while still running the old code, which seems like begging for weird database consistency issues.)

[antonpaquin]

Alright, then I'll shelf that for now.

For the purpose of setting up build and deploy scripts, would it be reasonable to assume the starting point: "we're SSH'd in to a clean ubuntu 18.04 with the repo cloned" ?

[toadworrier]

I like the conclusion in https://github.com/themotte/tildes/issues/18#issuecomment-653731912. My caveats

1. Not shell, use Python at minimum maybe even a static language like Go.
2. Ansible might then be the way to go -- the whole point of Ansible is to be a framework for SSHing into machines and running python modules to do idempotent sysadmin tasks.
3. There is a huge overlap in the use case between Salt and Ansible so it's annoying to have both.

As long as you are happy to not worry about (3), then I recommend starting with PR https://github.com/themotte/tildes/pull/16 and then improving the Ansible stuff in order to get rid of the provision.sh script.

[Deimos]

Hopefully this doesn't come off as patronizing, but I'd urge you all to remember that your end goal is setting up a small forum running on a single server. That's what the main Tildes instance is too, and over the two years of running it so far, I can't imagine I've spent more than a grand total of about 10 minutes just doing deployments "manually".

If you enjoy working on relatively complex devops-ish deployment systems for their own sake, definitely don't let me discourage you. But it looks to me like there's a lot of effort going into debating and worrying about solutions for problems you don't have yet, and almost certainly will never have.

[zorbathut]

Well, there's a reason this is under P1 and not P0 :V

One reason I'm concerned about this, though, is that I know myself well enough to know that I'm able to screw up simple instructions. And unlike you, I don't know the codebase inside and out; if we end up in a situation where things are broken it may be a painful process to get out of it. This isn't a timesaver, in my mind, it's a disaster-avoider; I'm not trying to avoid spending ten minutes doing deployments, I'm trying to avoid spending hours doing disaster recovery.

All that said, I do agree that dedicated programmers should probably be working on other things. It's the devops people that I've been pointing over here :)

[Zenchreal]

@Deimos those instructions worked a treat, thanks! Is it possible to skip adding tildes to sudo or will that break something?

Some additional steps I had to do on my system:

1. Manually add the nginx user
2. Copy prod.sls.example to prod.sls and change the hostname (this is something we added)

[Deimos]

Hmm, I can't think of anything offhand that needs tildes to be in the sudo group, so it should probably be possible to skip that. Almost everything that needs sudo/root should be getting done by Salt anyway.

Thanks for the notes about missing steps - I should write that up as a proper Docs page eventually, so I'll make sure to add those in as well.

[Zenchreal]

I recreated the deployment steps in an Ansible playbook in pull request #40

[zorbathut]

As per https://github.com/themotte/tildes/pull/40 I'm calling this Done Enough - it's actually pretty easy to get it up and running now.

Thanks, @Zenchreal !