Move primary deployment from Linode to Heroku

[jwm]

We should move our primary deployment from Linode to Heroku. We noticed recently that uWSGI hadn't picked up any changes for a few weeks, and Asheesh said he's to the point where he'd rather sacrifice a little cash for having someone handle running the deployed code.

[willingc]

@jwm Sounds like a good idea to me.

[jwm]

Still plugging away at this; converting the database data is proving to be a bit of a rabbithole, but that should be the last major thing to sort out before we can make the move.

[willingc]

@jwm Excellent. Selena Deckelman would be a good resource if the rabbit gets stranded in the hole ;)

[paulproteus]

Hi @jwm .

I noticed this issue has gone cold for a while.

I also wonder if we are trying too hard. Is there a way to use MySQL on Heroku? Can we get by with python manage.py dumpdata then python manage.py loaddata?

If you're too busy with other things, that's fine, but I don't think anyone else is tackling this so if you feel like looking into it again over the next N (1-8 weeks) that'd be glorious. And if other things in life take priority, that's fine too.

[jwm]

Hey @paulproteus, I had gotten pgloader working and most of the other pieces ready; sorry, this kind of got lost in everything I had to catch up with post-PyCon.

I set up a sample deployment at http://stormy-peak-6021.herokuapp.com/ (running code from https://github.com/jwm/oh-mainline/tree/jwm/heroku). I'd love it if someone could take it for a test drive and give the code changes a quick eyeball.

The gating factor right now is setting up a paid Heroku account with the sizing we need. I think a Hobby:Basic Postgres tier ($9/month) should be fine, running on a Hobby dyno ($7/month). If we need more workers than 512MB of RAM can sustain, we'll have to upgrade to a 1x dyno ($25/month) and scale up the dyno count.

Rough checklist/outline for doing the actual cut:

• any secure host:
  • ssh-keygen -f auto -P ''
  • heroku config:set REMOTE_REPO_SETUP_SSH_KEY="$(< auto)"
  • heroku domains:add openhatch.org
  • heroku domains:add www.openhatch.org
• linode.openhatch.org:
  • add auto.pub to ~/.ssh/authorized_keys, with `command="~/milestone-a/manage.py
  • mysqldump -uoh_milestone_a -p oh_milestone_a | gzip -9 >oh_milestone_a-$(date +%Y%m%d).sql.gz remote_command_check"`
• some other host with mysqld and postgres:
  • mysqladmin -uroot -p create oh
  • gzip -dc oh_milestone_a-$(date +%Y%m%d).sql.gz | mysql -uroot -p oh
  • createuser -DRPs "$USER"
  • createdb -O "$USER" oh
  • update mysite/settings.py for local postgres instance
  • USE_POSTGRES=1 ./manage.py syncdb --migrate --noinput
  • pgloader pgloader.conf
  • pg_dump oh >oh.pg
  • psql -d "$(heroku config:get DATABASE_URL -a openhatch-production)" <oh.pg
• Log into CloudFlare and add/update these entries:
  • CNAME www.openhatch.org to the Heroku app hostname
• Verify that git and svn repos are being created on linode.openhatch.org.
• Double check that everything is well with Travis, since it will now use Postgres for tests.
• Verify cron jobs on linode2 are running successfully.
• There's currently a non-revision-controlled job in root@linode.openhatch.org's crontab that does MySQL backups. Update that to pg_dump instead, install the Heroku toolbelt on linode2.openhatch.org, and put that cron job in mysite/config-files/crontab.linode2.
• Update the deployment docs.

[paulproteus]

(I'm replying to an offline copy of this github thread in my email, so I might be a few days out of sync)

Just to reply to the money part of this:

OpenHatch Foundation Incorporated is fine with paying a few more dollars per month (say, up to $50/mo, without that being a big deal at all) for the sites running on Heroku rather than our own Linode. If it's more than that, we can think harder, but anything under that, assume you can just get a "yes".

[paulproteus]

Also, a note about my scheduled this week:

I'm at this year's Debian conference for the coming week, so won't be able to do very much on this. I massively support other people doing things and then telling me, "Asheesh, do the following whenever you can, we've tested that it will result in good things." I'll be available on+after Mon Aug 24 if you need more bandwidth from me.

If you do a database migration & test run, you'll eventually need my help to cut DNS over. For that, I can give you the password to our CloudFlare.com account, or do it myself you ping me when you need it done.

Do please first test the new site by setting up your /etc/hosts line to simulate the DNS switch.

We're going to have to update some docs after the switch, and presumably we'll remove MySQL as being one of the supported databases for development, and use sqlite+postgres as the two supported databases for development.

[jwm]

I crossed off the remaining TODOs today and updated the checklist in my previous comment. I also opened https://github.com/openhatch/oh-mainline/pull/1693. Everything's looking good for the cut.

@paulproteus when you have a few minutes, would you create the Heroku app and give me access to it? Something like this should get us going with the dyno and Postgres addon we need:

heroku create openhatch
heroku sharing:add jwm@horde.net
heroku addons:create heroku-postgresql:hobby-basic
heroku addons:destroy heroku-postgresql:hobby-dev

[jwm]

@paulproteus Also, if you would send me our CloudFlare creds in some secure way (PGP-encrypted mail? I'm on the major keyservers as well as Keybase), I'd appreciate it.

[imported-from-roundup]

I believe I've done the Heroku setup stuff:

➜ oh-mainline git:(master) ✗ heroku create openhatch-production Creating openhatch-production... done, stack is cedar-14 https://openhatch-production.herokuapp.com/ | https://git.heroku.com/openhatch-production.git Git remote heroku added WARNING: Toolbelt v3.41.4 update available. ➜ oh-mainline git:(master) ✗ heroku sharing add jwm@horde.net ! Invalid arguments: "add" and "jwm@horde.net" Usage: heroku sharing

list collaborators on an app

Example:

$ heroku sharing === example Collaborators collaborator@example.com collaborator email@example.com owner

Additional commands, type "heroku help COMMAND" for more details:

sharing:add EMAIL # add a collaborator to an app sharing:remove EMAIL # remove a collaborator from an app sharing:transfer TARGET # transfers an app to another user or an organization.

➜ oh-mainline git:(master) ✗ heroku sharing:add jwm@horde.net Adding jwm@horde.net to openhatch-production as collaborator... done ➜ oh-mainline git:(master) ✗ heroku addons:create heroku-postgresql:hobby-basic Creating postgresql-dimensional-4168... done, ($9.00/month) Adding postgresql-dimensional-4168 to openhatch-production... done Setting DATABASE_URL and restarting openhatch-production... done, v3 Database has been created and is available ! This database is empty. If upgrading, you can transfer ! data from another database with pgbackups:restore Use heroku addons:docs heroku-postgresql to view documentation. ➜ oh-mainline git:(master) ✗ heroku addons:destroy heroku-postgresql:hobby-dev ! Can not find add-on with "heroku-postgresql:hobby-dev"

John, I believe that's everything you need Heroku-wise, too. Note I called it openhatch-production because openhatch seems to be taken.

[willingc]

@jwm Nice!

[jwm]

@paulproteus I just realized that Cloudflare wants you to log in with an e-mail address and password, but I don't know what e-mail address OpenHatch uses for its Cloudflare account.

Also, I think this is ready to do the actual cut. What kind of advance notice should I give (to openhatch-devel@ or #openhatch?), and are there any particular days or times where the site shouldn't be down for an hour or two while I get a consistent database snapshot and do the cut?

[paulproteus]

Hi!

I think asheesh@openhatch.org javascript:_e(%7B%7D,'cvml','asheesh@openhatch.org'); should work for CloudFlare username.

I would say, give ~24 hours of notice to devel@lists.openhatch.org and maybe Tweet something that we can RT as @openhatch and then pick any 2 hour window that works for you.

[jwm]

Ok, I think the cutover to Heroku went fairly well, though it took a bit longer than I'd hoped. The site should be up, the remote command execution for git/svn repo setup works, all the database data is there, etc.

@paulproteus: a few questions, since I realized doing the cut that the Apache configuration on linode.openhatch.org is pretty sprawling and has a bunch of what looks like cruft in it:

• The wiki hosted on linode.openhatch.org seems to be a going concern(?) since things like openhatch.org/contacts redirect to /wiki/Contacts. It looks like a MediaWiki instance had been grafted onto http://openhatch.org/wiki using an Apache Alias. I moved the wiki to http://wiki.openhatch.org, but the Django redirect I added for /wiki/* -> wiki.openhatch.org doesn't seem to be working, and when I go to http://wiki.openhatch.org, I get a redirect to openhatch.org.
• CloudFlare is doing SSL acceleration for www.openhatch.org, but it looks like we're not taking advantage of it. Even the old site on linode.openhatch.org wasn't using HTTPS for things like /account/login/old - am I interpreting this right?
• There are some Apache redirects, such as for things like /blog (hosted on WPEngine at blog.openhatch.org now?), /forum (forum.openhatch.org, which is running on linode.openhatch.org), etc. I'm not sure how much we care about these redirects, but there are a ton of them and it's hard for me to make heads or tails of what we should keep and migrate to Django redirects. Can you take a look?

I guess what this boils down to is... help?

Left to do/check:

• Check that e-mail sending works. I generated a bunch of 500 errors while testing, but never received any e-mail via monitoring-private.
• Verify cron jobs on linode2 are running successfully (move these back to linode, since we need to keep that host for the foreseeable future?).
• There's currently a non-revision-controlled job in root@linode.openhatch.org's crontab that does MySQL backups. Update that to pg_dump instead, install the Heroku toolbelt on linode.openhatch.org, and put that cron job in mysite/config-files/crontab.linode.
• Update the deployment docs.

[willingc]

@jwm Nice job documenting the migration. I created a milestone for Post Migration issues. I will wait to hear from @paulproteus and @ehashman if they wish to migrate your comments over as individual issues.

The website seems faster to load now once you get past the initial load :D

[ehashman]

I aggregated some of the broken link reports we were getting in #1701, looks like the wiki bit was already known now that I'm reading @jwm's comment above. Cheers!

[jwm]

Fixed the wiki and blog proxying.

[jwm]

@paulproteus looks like the openhatch-production app is running on a free dyno, and it's running out of memory. can you 'heroku dyno:type web=standard-2x' so we get some more memory?

[jwm]

I spent a big chunk of time poring over the nginx and Apache configurations on linode.openhatch.org. I took out large sections of the nginx configuration that aren't used any more or for sites that are now hosted elsewhere, leaving {git-mission,wiki,roundup-archive}.openhatch.org and openhatch.com. *.openhatch.org gets passed to Apache, and I wound up leaving the Apache configuration mostly alone since it's pretty sprawling and difficult to separate the wheat from the cruft. If you want to go back to the old nginx config, I init'ed a git repo in /etc/nginx before I made major changes.

I also enabled the 'Always use https' page rule at CloudFlare, which I think will replace the old nginx redirect from HTTP to HTTPS for all pages.

Additionally, I set up nightly Postgres backups at heroku (heroku pg:backups schedule DATABASE_URL --at '07:00 UTC'). I left the MySQL backups on linode.openhatch.org in place, since the Wiki and maybe other things use that MySQL server.

There are a few things that someone else should/needs to check or decide; either I don't have the access, or they're judgment calls that someone like @paulproteus should make:

• I couldn't enable a page rule at CF to automatically redirect HTTP->HTTPS for www.openhatch.org, since we've reached the maximum number (3) of page rules we can have. From my read of their URL patterns documentation, there's not a way to add a page rule for both *.$domain as well as the domain itself.
• https://openhatch-production.herokuapp.com/ works and CloudFlare's crypto setting is set to 'Full' (not 'Strict'), so I think connections between CloudFlare and our Heroku app are encrypted, but it'd be nice to have another set of eyes on this.
• Check that bug crawling is working on linode2.openhatch.org and that it's updating the Heroku postgres database. I set DATABASE_URL in local_settings.py on that host, so I think everything should be fine..
• The wp-cron job on linode.openhatch.org is maybe not necessary? If it's necessary, I'm not sure if it's working, since accessing that Wordpress cron endpoint yields an empty page. I haven't hosted WordPress in a long time, so I don't know if that's normal.

I think that should be everything, other than the bugs I've opened (#1707, #1708, #1709) and the last step of the git mission, which I missed applying remote repo server support to.

[Lucas-C]

Hello.

With OpenHatch winding down, I guess this issue won't be solved: https://blog.openhatch.org/2017/celebrating-our-successes-and-winding-down-as-an-organization/

It's a pity, the /search page was awesome. Now all links are 2 years old and often point to closed issues.

Do you recommend any alternative ?

[willingc]

Hi @Lucas-C,

Thanks for the kind words. There are a few alternatives that I know of:

• codetriage.com
• use the new GitHub feature that indicates that projects have issues needing help
• perhaps see what the libraries.io folks have (it's a cool project but I don't know if they track issues needing help)

[Lucas-C]

Hey, thank you very much for your quick answer !

I couldn't find info about this new Github feature, but in the meantime I discovered http://up-for-grabs.net & http://www.firsttimersonly.com

I'm going to check https://www.codetriage.com, it looks great. It doesn't have your "mentor" feature though :(

Regards

[willingc]

Hmm... I thought there was a blog post on the new GitHub label "Good for New Contributors" and the new line in the projects repo header:

There's also this github showcase: https://github.com/showcases/great-for-new-contributors

[Lucas-C]

Thanks !