No autobuild on cutorch branch?

[hughperkins]

Seems like it'd be good to have an autobuild on cutorch branch? having a dedicated titan or ec2 g2 instance seems overkill :-D but you could set up an ec2 instance to start whenever there is a commit, connect to a jenkins, have the jenkins already have queued a build job for it, and shut down the instance after the test, for example.

This would have a couple of advantages:

• can check things build, easily
• can check tests pass, easily

[soumith]

this is a good idea and on my list forever, just haven't had the time to figure out how to do all of this....

[hughperkins]

I guess a key prerequisite is to have a suitable machine or ec2 iamuser. I have a bunch of scripts around to do things like, start an existing ec2 instance, and so on.

[hughperkins]

... and so, if you provide me an appropriate iamuser account/keys and/or we create some sort of private/shared repo for jenkins scripts etc, I might help with this. Maybe. Possibly. No guarantees. But maybe :-)

[hughperkins]

You know, nimbix instances make doing this really easy. Since:

• per-second billing :-O
• spinup times of ~ few seconds
• REST api to spin up an instance, run a single command, and get the output
• I made some python scripts using their REST api that let you run scripts synchonously, from your command prompt, but they actually run on their instances https://github.com/hughperkins/nimbix-admin#scriptpy---run-a-single-batch-script-then-exit

[soumith]

If you set it up via your personal account, i can send you money for this every month

[soumith]

I just dont know how to setup the amazon thing. Another option is we do a quick screenshare and i can setup the amazon thing with my billing that you can use for this...

[hughperkins]

Ok. I think we basically need three parts:

• amazon ec2 t2.micro instance will hold an 'always-up' jenkins instance. Cost of an always-up t2.micro ~10usd/month
• a way for jenkins to communicate to and from github, which is:
  • from github to jenkins, is configured as a service in github repo, that points to jenkins. This will trigger jenkins builds
  • from jenkins to github. I set up a prototype for how this works at https://github.com/hughperkins/test-checks/blob/somebranch/post.py Example pull request with status checks https://github.com/hughperkins/test-checks/pull/1
• nimbix for running the build and unit tests. They have Titan X instances and per-second billing. It's trivial to spin up an instance, to run a single script (eg build script, or unit test script), and let it spin down again. If we have ~200 builds a month (10 merges * 10 commits per merge * 2 repos (cutorch, cunn), and we build on a low-cost machine, run the tests on the dual titan instances, then ... cost per build/unit test would be around (15/60 hours * 0.36usd/hour ~= 0.09usd) for build and (5/60 hours * 3usd/hour ~= 0.25usd) for tests, or ~70usd a month.

One conundrum which occurs to me: you and I have both set up our travis tests only on our distro repos, not on the actual cutorch/cunn/cltorch/clnn repos. Since the repos cant actually be tested in isolation. Personally I actually update my distro-cl for all changes to the underlying repos https://github.com/hughperkins/distro-cl/commits/distro-cl , but that's not really how it works for cunn currently. So ... thoughts on this?

[soumith]

@hughperkins

The costs look pretty good, no worry there.

I've setup travis tests not just on distro, but on individual nn, torch7, image, nngraph etc.

We follow the same for cutorch/cunn?

[szagoruyko]

this is great. I have travis config for cutorch https://github.com/szagoruyko/cutorch/blob/master/.travis.yml, cunn is trivial to modify and add actual GPU tests

[hughperkins]

First part: create ec2 t2.micro instance: https://github.com/hughperkins/torchunit/blob/master/ec2instance.md

[hughperkins]

nimbix https://github.com/hughperkins/torchunit/blob/master/nimbixsignup.md

But as I wrote it, I realized that I dont really want to have your apikey, on the whole. Too dangerous if it somehow leaks out.

So, as I was writing, I rethought a bit, and think maybe no-one except you should have login to the jenkins box, and jenkins box will run from scripts in a github repo. You can put your nimbix apikey on that box, noone else can see it. We are free to modify the jenkins instnace config by pushing pull request to the repo containing the jenkins instance installatin/configuration scripts. Then your api key is safe.

[hughperkins]

The scripts at https://github.com/hughperkins/torchunit are enough to install and start jenkins now, including creating a self-signed https cert :-)

To install:

• ssh into ec2 box
• paste the following:

wget https://raw.githubusercontent.com/hughperkins/torchunit/master/installjenkins.sh
bash installjenkins.sh

To run:

• ssh into ec2 box
• paste the following:

torchunit/runjenkins.sh

To access the (entirely unsecured for now ... ) jenkins, navigate to:

https://52.1.2.3:8443/

.... where 52.1.2.3 is the address of the ec2 box

[hughperkins]

(you'd probably want to fork my repo, so you can control pull requests to it, I would imagine :-) )

[hughperkins]

hmmm, tweaking some stuff...

[hughperkins]

Enabled security now. Just some basic security so by default people only have read-only access, which sounds pretty good for now. The install procedure has changed:

in the instance ssh, paste and run:

wget https://raw.githubusercontent.com/hughperkins/torchunit/master/bootstrap.sh
bash bootstrap.sh

2. copy torchunit/config.yaml.templ to torchunit/config.yaml , and set a jenkins user password in it

3.

bash torchunit/installjenkins.sh

Run procedure unchanged, just do, from the intsance:

bash torchunit/runjenkins.sh

So far, no jobs or anything, but at lesat its secured. There are ways and means of creating jobs, ie/eg http://docs.openstack.org/infra/system-config/jjb.html

Example jjb jobs:

https://github.com/hughperkins/DeepCL/blob/master/jenkins/jobs.yaml

[hughperkins]

Actually... it occurs to me, can we just directly drive nimbix from travis? I guess we can ???

[hughperkins]

Seems like if we put the nimbix apikey in travis repo settings, it would only be available to pull requests submitted by eg soumith. other pull requests wouldnt see the apikey. which makes sense. but limits the usefulness a bunch...

https://docs.travis-ci.com/user/environment-variables/

"Encrypted environment variables are not available to pull requests from forks due to the security risk of exposing such information to unknown code."

So, a webservice, on a trusted machine, is perhaps the way to go. Might not need to be a full jenkins thing though. Could just be a service that:

• is triggered by eg travis, calling into some webservice api endpoint
• checks that the callee was travis, and not some random ip address (how? is this possible/easy?)
• doesnt just blindly do whatever the caller tells it, but accepts just like one or two parameters, ie:
  • commit hash for the incoming pull request, and ...
  • nothing else?
• based on this, it already knows itself, is locked to, the cutorch repo, and knows how to run the build etc (or: it could clone that fork, and run eg .nimbix.yaml from that fork. This is probably not super unsafe, but not really safe. probably it should pull down a separate 'master' branch, and run .nimbix.yaml from that? (or .unit.sh plausibly, to be independent of vendor)

[hughperkins]

Apparently we can detect travis boxes as https://docs.travis-ci.com/user/ip-addresses/ Whether it's worth doing this, since anyone can run any build on travis is an open questoin. The real security comes from the fact that the secure webservice wont execute arbitrarily code, simply pull requests of cutorch. Which is not theoretically non-arbitrary in itself. Maybe what we want is something like that magical 'test this please' phrase I saw somewhere, where if a nominated repo admin said "Test this please", the build bot would run :-O :-)

[hughperkins]

(seems there is a jenkins plugin for the 'test this please' functionality https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin )

[hughperkins]

Seems it is possible to raise a support request to fund a nimbix account via Paypal, prior to using it. This would then presumably limit the upper bound cost of some runaway usage to however much one funded it for. They promise to terminate all instances once such funding has been used up, though I suspect this is kind of a manual process for now, on their part.

[hughperkins]

(I've made a systray icon for nimbix, that shows any instances we have started https://github.com/hughperkins/nimbix-admin/blob/master/ubuntuindicator.py This means that if someone does somehow get the apikey, and start a zillion intsances, we'd notice pretty quickly. I built this after leaving one of the dual titan instances running overnight :-D )

[hughperkins]

Ooo... I found an example of someone using 'test this please'. It is ... Tensorflow :-P https://github.com/tensorflow/tensorflow/pull/2556#issuecomment-222893778

Edit: hmmmm, I wish github would warn me it's going to add a link, or ask me if I even want a link ... :-/

[soumith]

Okay, off to a good start,

$JENKINS_IP = "50.17.86.9"

[soumith]

@hughperkins installed jenkins, and gave access to the box to your private keys (that you sent to me)

[soumith]

next step, i'm signing up for nimbix.

[soumith]

i've signed up for nimbix, added you to the "Team" (invited your gmail address), and added in my payment details. I tried to follow you on the instrunctions at the end of the Nimbix signup document, but you list two possibilities there, so got confused. What next? What else do I need to do?

[hughperkins]

Cool! :-) Started looking at next steps in https://github.com/hughperkins/torchunit/tree/master/jenkins.md

[hughperkins]

As far as the Nimbix API key, I'm not sure I'd be entirely comfortable with having my own nimbix apikey on a shared server, therefore by extension, I wouldnt be entirely comfortable with putting your own key, or a proxy of such (eg via Nimbix team structure), on a shared server.

I think we should have a wrapper webservice around the nimbix api, on a standalone ec2 instance, which wraps the nimbix api, and does things like:

• only can launch read/write instances, always of the same image => this enforces effectively that can only run one instance simultaneous, (cf read-only instances, which can be launched arbitrarily many times, simultaneously, as far as I know)
• responsible for having the apikey, jenkins doesnt have it
• has a shared secret key with jenkins. this shouldnt be publically known, but doesnt have to be fort-knox secure, since it only gives access to the wrapper service
• could have ssl with connection to jenkins, but not essential, since again, only gives access to the wrapper, not to the apikey itself
• sets each job to run for a maximum of eg 40 minutes, and then shut itself down (this is part of the nimbix api)
• only runs some very specific script, from some specific repo, albeit with a (cleaned) parameter, being probably the git commit hash
  • eg might run a script from eg torchunit repo, called eg nimbix-bootstrap, on the launched titan x instance, and it will pass in the git commit hash as a parameter
  • the parameter should be cleaned to ensure it only contains a-f and digits 0-9, and no more than 40 characters long

I can look at writing such a wrapper service, probably in Flask, and add it in eg into torchunit repo, (or possibly into some generic repo 'nimbix-wrapper')

[soumith]

no need to write a wrapper service and stuff. Just put my API key on the jenkins server, it's fine. Only me and you are sharing it, and nimbix has a cap on my account.

[hughperkins]

Only me and you are sharing it

Well, for now. But would probably be good if other people can admin it too, I would think. I have written a wrapper at https://github.com/hughperkins/nimbix-admin/blob/master/wrapper-service/nimbix-wrapper.py It has all the security detailed above, and in addition is locked to the ip address of the jenkins box (specified in https://github.com/hughperkins/nimbix-admin/blob/master/wrapper-service/config.yaml.templ ) I'll provide some instructions/script for setting this up, soonish.

[hughperkins]

(Also, the attack surface of an instance containing a small tiny lightweight surface is much smaller than a big jenkins box, I would think)

• documentation on how to set up wrapper box https://github.com/hughperkins/torchunit/blob/master/wrapperinstancesetup.md
• some simple security analysis here: https://github.com/hughperkins/torchunit/blob/master/wrapperinstancesetup.md#security-analysis

[hughperkins]

(basically, the situation I want to avoid is:

• somehow someone, could be you, could be anyone, launches a bunches of instances using your api key
• you know I have the apikey
• I have to somehow prove/convince it wasnt me, which would be hard to do

Easiest way is, I simply dont have the apikey, then I dont have to handle this situation as and when it arises...)

[soumith]

no problem, i just didn't want you to do extra work when it was mostly a hypothetical. I mean, I trust you to not care whether you have an API key or not. will start taking a look at it wednesday-ish, at this day-long conference today.

[apaszke]

@hughperkins btw, if you really want to be secure you probably need to make it a challenge-response authentication, rather than a single request. If someone sniffs the shared key, they can simply spoof packets as coming from a Jenkins box IP. They don't need your response to spin up jobs, so the IP check doesn't do much right now.

Even simpler, you could make it work more like CSRF. First Jenkins requests a temporary security token (+ it has IP checked), and then it submits a work request, that has to contain the exact same token, which is invalidated immediately. No shared secret required.

Adding https://letsencrypt.org to the service can minimise the chance someone will see the secret/token in transit. I can help if you need any assistance.

[hughperkins]

they can simply spoof packets as coming from a Jenkins box IP

I think they'd struggle to spoof a tcp/ip connection? They'd get as far as sending 'SYN', server sends 'ACK', and then ... ? Unless they control a router in between.

Adding https://letsencrypt.org to the service can minimise the chance someone will see the secret/token in transit

Yes. Actually, I wrote a 'howto' for this, targeted at jenkins usage, for those like me who dont fancy letting lets-encrypt install libraries all over the place https://github.com/hughperkins/howto-jenkins-ssl/blob/master/letsencrypt.md I've never dabbled in enabling it for flask though.

I can help if you need any assistance.

Sure, sounds good :-)

[apaszke]

As far as I remember you don't have to install much. You can obtain the certificates through https://gethttpsforfree.com. I didn't check its code, but it's open source and is listed on let's encrypt website, so it's probably ok.

For the flask part, it seems to be quite easy too: http://flask.pocoo.org/snippets/111/

[apaszke]

On second thought, it's not a public service, so letsencrypt is an overkill. Self-signed certificate should be ok.

[hughperkins]

On second thought, it's not a public service, so letsencrypt is an overkill. Self-signed certificate should be ok.

Ok. Yes, and shared secret just lets them schedule a set of cutorch unit tests anyway. I'm sure that will be fun for them :-D Maybe I'll draw a picture of the architecture I'm imagining.

[hughperkins]

https://docs.google.com/presentation/d/1Jiddfqg3yko-LLE3oURfB-yEOdOFi3mDI4Hp9v5B0sk/edit?usp=sharing

[hughperkins]

[apaszke]

Thanks! You did describe it clearly earlier, but that's a nice overview.

I just meant that people who can see the shared secret are also the ones that probably would be able to do some fun IP spoofing, and adding self-signed certificate to flask should be quite easy.

Nevertheless, yeah, it would only let them burn some money on unit-tests (or on arbitrary stuff, if they send a PR that completely changes the tests, and do "Test this please" with their own triggers).

[hughperkins]

Nevertheless, yeah, it would only let them burn some money on unit-tests (or on arbitrary stuff, if they send a PR that completely changes the tests, and do "Test this please" with their own triggers).

On the subject of Jenkins, 'test this' plugin needs 'push' access to the repo. This sounds quite powerful to me, so I might check to what extent it is subject to checks such as, cannot push to 'master', and cannot force push to master.

(Original message, can skip too long:

Nevertheless, yeah, it would only let them burn some money on unit-tests (or on arbitrary stuff, if they send a PR that completely changes the tests, and do "Test this please" with their own triggers).

You mean, if someone has access to the Jenkins? So, in related news, yesterday, when I started looking at putting in place 'test this please', it turns out you need to grant 'push' access to jenkins, in order to enable this. I dont use this module on any of my own jenkins currently. What I tend to do to date is:

• set up automatic build and test of any commits to my own repo
• if someone submits a PR, I push their commit into my own repo (ie in a separate branch), which will trigger the tests, and see the result pass/fail

The 'test this please' I think basically automates this second step, as far as I can tell, though I confess I havent looked in detail yet. It does however imply giving 'push' access to Jenkins. Thinking about what is possible with such access, I came up with:

• delete entire branches => probably loads of people will have copies of these, so not a major issue if it occurs?
• force push entirely changed histories, or slightly changed histories (eg 1 additional commit, containing some payload) => probably would be noticed quite quickly, since people would get the message about 'force pushed', and maybe check a bit (reflog would allow inspection of before/after I think?)
• add a brand new commit, to HEAD. This wouldnt trigger a message, and if done in the name of someone who:
  • isnt monitoring closely, but
  • has done a bunch of commits in the past

...might not be noticed immediately.

... however... since it's using a separate github account, presumably subject to github standard repo permissions, maybe:

• it cannot push to 'master', as long as master is configured as a 'protected' branch
• including cannot force push to master
• ... and this would therefore rule out pretty much all three attacks above? (It could modify branches, but these would still have to go through PR anyway, so maybe not a major issue?) )

[hughperkins]

step 1 of checking limits of limits on possible future torchbuildbot account:

• pushing directly to master blocked, as long as master is protected https://gist.github.com/hughperkins/9149c27461c7b82bbd68bf68b4fe946c#file-push-master-blocked-L63

Next up: need to confirm that said permissions are sufficient for 'test this' to work ok

[hughperkins]

Hmmm, cant figure out how to configure the webhook to connect from github into jenkins pull request biulder plugin. I just get:

... all the time. It seems that most jenkins-side stuff assumes one has a github account with admin access, that one can put into jenkins, and it will set up the hook for you. So, documentation on setting up the webhook in github is pretty sparse. What I tried (on my own jenkins/github repo), is to put a shared secret in the jenkins system configuration at:

... and use the same secret value in the github webhook configuration

I've been poking at this for an hour or so, cant seem to figure out how to get this working?

It's possible to set the jenkins build to poll periodically, eg every 20 minutes, but that'd be a bit annoying to use probably?

[hughperkins]

Raised an issue at https://github.com/jenkinsci/ghprb-plugin/issues/378

[hughperkins]

Ah, figured out the webhook bit, thanks to http://stackoverflow.com/questions/7427557/jenkins-and-github-webhook-http-403/7431548#7431548

[hughperkins]

Whee, I got test this please working :-) (on a test repo/jenkins). Bunch of hoops to jump through, I'll document them soonish...

[hughperkins]

(And note that the 'test this please' plugin also handles setting status on the pull request, and so on; seems quite fun :-) )

[hughperkins]

(Hmmm, wow, thats scary, I was dabbling in providing anonymous access to the build logs of my own jenkins, and noticed that all my enviornment variables are exposed :-P seems like launching jenkins with env -i might be a good idea... https://github.com/hughperkins/torchunit/commit/4d7561a04f8a3a69f7ea3ebb92945f988b0d3718 ; also, EnvInject plugin is plausibly not ideal on internet-facing jenkins... )

[hughperkins]

So, I experimented with removing 'push' access to the bot (on my test repo/bot/jenkins)

• the bot will still do a build
• it will report the build started/finished as comments into the PR (but without a link to the build log)
• it wont update the PR build status

Without push access:

With push access:

I think that having the PR build status updated is a pretty fundamental really useful aspect of having autobuild, so I think the bot should have push access to the repository, just make sure that at least 'master' is protected, and cannot be force pushed to.

Thoughts?