search

lobsters / lobsters-ansible

Ansible playbook for lobste.rs
ISC License
79 stars 26 forks source link

start unicorn at boot #7

Closed pushcx closed 6 years ago

pushcx commented 6 years ago

This is currently a manual step of, as the lobsters user, running:

cd /srv/lobste.rs/http
bundle exec unicorn_rails -c config/unicorn.conf.rb -E production -D config.ru

I don't think the box has seen a clean reboot yet and it'd be great to test this while the config is still fresh in our minds.

jstoja commented 6 years ago

Do you want to solve this by adding lobsters' unicorn server as a systemd service?

pushcx commented 6 years ago

I think that would work fine for starting and restarting (#3), yeah.

jstoja commented 6 years ago

Hey @pushcx You added this file in commimt 72a1100f2568bed211ad8ba65c0eecc52f2ddd07. you can close this one too.

pushcx commented 6 years ago

I added the unicorn config, but there isn't a service to start or restart it. I've been manually doing it.

Previously I did:

cd /srv/lobste.rs/http/; pkill -QUIT -f 'unicorn_rails master'; sleep 1; bundle exec unicorn_rails -c config/unicorn.conf.rb -E production -D config.ru

This has two disadvantages: the site is down for a second (nginx throws 500s), and if there a unicorn worker hasn't exited after a second (probably because it's serving someone on a slow connection) the new unicorn master sees it and refuses to start.

So I started using this: pkill -HUP -f 'unicorn_rails master'; echo $?

And the disadvantage is that it doesn't preload the app, so the first connections to each worker dogpile and are very slow, taking 15-20s as the Rails app is loaded.

What I want is that the new app server process is started and the old master process hands over the socket to it, killing off its workers as they finish or after some timeout, so no connection gets a 500 or long load. I don't understand why this isn't the default behavior of unicorn and I don't see how to get there from here. This may mean we want to replace unicorn with puma or passenger.

pushcx commented 6 years ago

With ca95915786c23712e3250ced410dd72c88256e6a and b6fc943fc47ca2cb088004825dd1733bf476df30 unicorn is now properly reloading when it receives USR2. New workers start with code preloaded and old workers exit as they wrap up their requests. This has smoothly handled a bunch of deploys in the last week and is such a relief.

We do still need config to start unicorn as the server comes up, so the issue is still open.