Trigger builds from GitHub webhooks

[cobyism]

Right now the only way to trigger a build is via DCIY’s web interface itself, but ideally builds should be triggered each time a push is made to GitHub for a project.

Things to consider:

• Do we automatically set up new projects if a payload is received?
• Making DCIY accessible from the internet may be hard for some people to work out how to do—documentation here will be important.
• If, people will be putting DCIY up on the internet publicly how to protect DCIY from malicious POSTs, or from people trying to look at it who shouldn’t have access.
• How DCIY knows about its own location on the public internet, and how to structure URLs in commit statuses (/cc #2) so that both the person running DCIY and other people collaborating on the project can all access build output etc. easily (going to a public URL for the project running locally may not resolve DNS and so forth)
• Can we automate the setting up of webhooks? Do we need API access to do this? do we just show people a webhook URL to copy/paste from DCIY with a token in it and explain where people should go in GitHub to set up the hook?

[smashwilson]

The hardest part I see here implementation-wise is working with setups behind a NAT router. If you don't have a port forwarded, there's not much you can do to get notifications. Fall back to polling, maybe? Ick. I'd thought about trying to wrangle something with hooks in your local dev repositories, but there's no "post-push" hook in git.

It shouldn't be too tough to automate though: on Project creation, find our external IP, build a URI with our port and callback path, and add a web hook with the GitHub API if one doesn't already exist. Do the same thing in a Rails initializer for existing Projects. If we want to be extra nice, deregister it in an at_exit hook so you don't get tons of failed postbacks on GitHub's end while your server isn't up :wink:

Note that this does depend on some mechanism for managing GitHub auth tokens, which we'll need anyway to interact with private repositories.

[smashwilson]

Also, automatically setting up a project when getting a payload would be dangerous. Anyone who finds your instance could get you to run arbitrary code! :warning: :warning:

[cobyism]

working with setups behind a NAT router

Yeah, I don’t really want to go out of the way to help people set this up. People who know they want to do it will probably know how to it anyway (have a dyn-dns provider, hook up nat at the router to forward the right ports etc). I just want it to be possible to trigger builds via the webhook payload and let people figure out how to make the endpoint available to GitHub’s webhook posting.

on Project creation, find our external IP, build a URI with our port and callback path

One option might be to simply specify a PUBLIC_HOSTNAME environment variable which you can set to something like http://my-dynamic-ip.provider.com:8080 which can be used when showing someone what to copy/paste into the GitHub webhook.

this does depend on some mechanism for managing GitHub auth tokens, which we'll need anyway to interact with private repositories.

I actually don’t want this to be something this app has as part of its scope—at least not to start with—that’s the whole idea behind the "Do CI yourself" concept. I really want the shelling out to just be done with your regular shell username and depend on having your own access to private repos set up correctly (i.e. having your ssh keys set up and so forth). The less we can have this app be responsible for (auth tokens, oauth app clients/secrets, ssh keys, automatic webhook setup), the bigger the learning barrier to setting up a simple local CI server you can run for personal projects, so I want to avoid as much of that as possible.

[smashwilson]

I just want it to be possible to trigger builds via the webhook payload and let people figure out how to make the endpoint available to GitHub’s webhook posting.

Okay, I'm on the same page now. Three cheers for simplicity and limiting scope! :beers:

I really want the shelling out to just be done with your regular shell username and depend on having your own access to private repos set up correctly (i.e. having your ssh keys set up and so forth). The less we can have this app be responsible for (auth tokens, oauth app clients/secrets, ssh keys, automatic webhook setup), the bigger the learning barrier to setting up a simple local CI server you can run for personal projects, so I want to avoid as much of that as possible.

Makes sense. So a typical run might actually be:

ssh-agent bash
ssh-add
# Enter ye password
script/server

That reminds me, should we be using ssh URIs in Project#repo_uri?

[cobyism]

So a typical run might actually be: [ssh stuff]

I would love to not even worry with any specific ssh stuff, since we get into problems with interactive shells requesting passwords and different access environments between shells etc. I simply want to have DCIY presume that people have keys/access set up themselves on the command line, so the app basically assumes that simply shelling out and calling git clone <whatever> will work the same way it would if the person manually cd’d into a directory and started executing commands as themselves from their own regular shell.

should we be using ssh URIs in Project#repo_uri?

Smart HTTP is a much better option IMHO. It’s less likely to be blocked by anyone’s firewalls, and is much easier to set up/understand since you can literally copy the URL out of your browser’s address bar when you visit a GitHub repository and it will work as a clone URL.

[smashwilson]

I simply want to have DCIY presume that people have keys/access set up themselves on the command line, so the app basically assumes that simply shelling out and calling git clone will work the same way it would if the person manually cd’d into a directory and started executing commands as themselves from their own regular shell.

Right right -- I wasn't proposing that DCIY would script the ssh commands, but that a user would run them in the same shell before launching the server. Basically I think we're agreeing :grinning:, but I was thinking in git-over-ssh terms by default instead of smart-http terms. (At work, getting https cloning to work properly means a ton of weird openssl incantations and libcurl configuration to get git to present certificates that are signed by the right CAs and identify you properly, so I've learned to go the ssh route first; here in the real world, I think you're right that https is the better option. I hadn't thought about firewalls!)