General improvements and saving notes [v0.4dev]

[avacariu]

General improvements:

• database is created and migrated at runtime, removing a deployment step and allowing automatic schema upgrades
• indexes can be built inside docker containers, removing the need for extra software as well as ensuring everyone's running the same version of everything
• provisioning a CentOS 6 server to run this code is now within a script, so you don't have to run all the commands manually from website.md. It's safe to run this script multiple times.
• new script for generating CA, Server, and Client SSL certs and keys so that we will be able to deploy to the remote Docker daemon without having to SSH into the server
• a Makefile now allows you to just make deploy or make dev to build everything and run everything (as well as targets to remove containers and images), so we don't have to remember all the different commands, or look them up in the website.md
• we are now back to a single docker-compose.yml because Docker Compose doesn't yet have all the features necessary for properly managing multiple yamls (e.g. you can't inherit anything you want). Once they add build-time environment variables, we'll be able to move more stuff into environment files (already exist in Docker, just missing from Compose: docker/compose#2163)
• database schema migrations now work properly; it now detects that we're using SQLite, so it uses batch mode, and it's now fairly straightforward to create a database, generate migrations, and apply them. It's not completely automatic -- it detects python-social-auth models incorrectly as to be deleted -- but it's pretty easy to just open up the migration file and delete those lines (I'm still working on getting it to detect those models, and I think I've got an idea of how to do it, but it's not in this PR).

New features:

• You can now save text notes. The UI only allows for a single note, but the backend + database allow managing any number of notes. Just so that we have something for this feature, it can only handle text, but it's easily extensible to other types.

This update will require removing all existing images and containers (make remove-images) because the current version of the database isn't stamped with a version ID, so this code won't be able to apply the migrations.

Note: The [v0.3dev] in the title is the tag I've used for the last commit in this PR so that we can keep track of when stuff in develop was stabilized and is ready for a merge to master, and also so that we can keep the develop branch long-running. Annoyingly enough, you can compare a tag and a branch in GitHub, but you can't create a PR based on it; for whatever reason, PRs only work between branches.

[anoopsarkar]

Is this ready to merge?

[avacariu]

It should be ready; I updated the tag to v0.4dev.

To deploy, it requires deleting the current images/containers and rebuilding, but that's all in the makefile now, so it should just be:

$ make remove-images
$ make dev

The target is called dev just because the deploy target is supposed to deploy to a remote docker daemon, and has some stuff about keys in it. The main server isn't set up for this yet.

One of the commits in here also switched back to using port 8080 so that we can use Apache as a reverse proxy for this site and others.

[anoopsarkar]

Is it worth doing all this deploy to remote container business? It seems complicated and brittle and does not save all that much effort, does it?

[avacariu]

I don't see why it would be brittle. Docker Compose communicates with the Docker daemon over HTTP anyways, and this is just exposing the port and adding SSL certificates.

In terms of saving effort for us, I don't know. I just wanted to test this out to see how it goes (not on the main server), but it's certainly not necessary for deploying what we've currently got; that's why I added two (basically identical) make targets. make dev is all we need right now.

[anoopsarkar]

Should we go ahead and merge this for testing? When we deploy it might be good to stick with port 80 as we sent out emails to several people with the default URL (without 8080).

[avacariu]

All this stuff is functioning properly on the staging server.

In terms of the port, we should just get Apache to act as a reverse proxy and listen on port 80, but leave Nginx within docker listening on 8080 (and probably close port 8080 from the public), so that multiple things can be hosted on the same machine on port 80.

[anoopsarkar]

ok. sounds good. can you update website.md to add the instructions on how to configure apache appropriately. add the text of the conf file(s) so that I can just copy/paste them in to get it working.

[avacariu]

It's documented now.