fix(boot.go,pkg/*): implement a deleted app cleaner

[arschles]

Overview

This PR introduces a background goroutine that polls the Kubernetes API for the list of namespaces and compares those names with those in its local git folder. The cleaner then deletes all directories that exist locally but not in the namespace list.

Testing Instructions

To test, follow these steps:

1. Create an app and git push to it from repository A
2. deis apps:destroy -a your_app
3. Wait at least the polling interval (the default is 1 second)
4. Re-create an app of the same name from repository B (so that the deis tool creates the same remote in repo B). Make sure repository B has a different set of refs from A
5. git push from repo B

The push should be successful and the app should run properly.

Still TODO:

• [x] Lock repositories in-memory while they're being deleted. The SSH server should reject pushes to repos that are currently being deleted. Depends on #170 for the RepositoryLock primitive in order to achieve this locking
• [x] Make the cleaner's poll duration configurable (it's currently hard-coded)

Notes:

• This PR also introduces global locks on git pushes, effectively removing some of the functionality in #170, but still preventing concurrent git pushes. We may want to hold on this PR in order to get back to full feature parity with that introduced in #170
• Assuming the previous were fixed, this PR could still introduce an edge case that can impact performance negatively. See https://github.com/deis/builder/issues/192 for details and a proposed solution.
• There's a polling window for the cleaner. If a user deletes an app, re-creates it, and pushes a different repo within that window, they'll see errors. See https://github.com/deis/builder/issues/193 for details and a proposed solution.

Fixes #167

[bacongobbler]

Seeing my logs being spewed full of messages:

Cleaner found no git directories to delete
Cleaner found no git directories to delete
Cleaner found no git directories to delete
Cleaner found no git directories to delete
Cleaner found no git directories to delete
Cleaner found no git directories to delete
Cleaner found no git directories to delete
Cleaner found no git directories to delete
Cleaner found no git directories to delete

Then I realized that debug is turned on by default. Works for me!

[arschles]

Note that I've manually tested this and verified it works on GKE

[mboersma]

Tested locally on a kube-solo cluster according to the above instructions. Everything went swimmingly. :+1:

[smothiki]

Tested Just now working

[gerred]

Tested on GKE as well. :+1:

[bacongobbler]

LGTM x 4

[krancour]

Any thoughts of a similar host-level trashman? Unused / old Docker images can fill up a disk. We should look into running something like Sherdock in a daemonset in the future.

[arschles]

@krancour not sure what you mean by host-level trashman. Can you file an issue with details? If it's something to clean up docker images I'm all for it, but the Sherdock readme doesn't provide much detail.