Use hubot to start GCE Jenkins Slaves

[winhamwr]

• Client library tutorial

[winhamwr]

this is the authentication method we must use. JWT with a service account.

• [ ] Figure out how to base64 encode/decode to push the value to an environment variable
• [ ] Figure out what scopes are necessary

[winhamwr]

Ideas for future enhancement

• Add specific tags to the VMs and only use instances with the tag when determining which zones have instances (since someone might have other types of VMs in the project)
• Even if there are no VMs in a zone, attempt to create 1 VM in a zone (which doesn't count towards our desired machine count) in order to discover when that zone is back to being not busy
• Use the GCE_MACHINE_COUNT only as the initial value and pull the real desired machine count from the number of downstream jobs. Store that in the brain.
• Before launching machines, check if there are already jenkins workers idle and no jobs in the queue. Only launch enough machines to fill the difference. (maybe this isn't worth it, though, because those workers might idle out before the pstat_ticket job finishes and spawns jobs for them)
• Figure out how workers can tell the master that they're shutting down, so that we don't need to do manual cleanup
• When a VM creation operation completes before timing out, handle the error cases for the operation. We hit a quota limit with a test operation and future-us should be able to dive the logs to figure out what's up.

TODO

• [x] Convert to coffeescript
• [x] Build a new worker disk image that actually does idle termination

[winhamwr]

It's probably not work using a callback on the VM creation. About half of them don't complete the specified time and it doesn't really give us any useful information, either way.

[winhamwr]

They're alive!

2015-09-11T23:28:59.382906+00:00 heroku[api]: Deploy a8a6ac3 by wes@policystat.com
2015-09-11T23:28:59.383023+00:00 heroku[api]: Release v213 created by wes@policystat.com
2015-09-11T23:28:59.716368+00:00 heroku[web.1]: State changed from up to starting
2015-09-11T23:29:02.546824+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2015-09-11T23:29:03.244955+00:00 heroku[web.1]: Starting process with command `bin/hubot -a hipchat -n Hubot`
2015-09-11T23:29:04.254630+00:00 heroku[web.1]: Process exited with status 0
2015-09-11T23:29:05.398303+00:00 app[web.1]: npm WARN package.json hosted-hubot@2.5.1 No license field.
2015-09-11T23:29:08.364331+00:00 heroku[web.1]: State changed from starting to up
2015-09-11T23:29:08.710147+00:00 app[web.1]: [Fri Sep 11 2015 19:29:08 GMT-0400 (EDT)] INFO Connecting HipChat adapter...
2015-09-11T23:29:09.154681+00:00 app[web.1]: [Fri Sep 11 2015 19:29:09 GMT-0400 (EDT)] INFO Connected to hipchat.com as @HubotBotson
2015-09-11T23:29:09.234363+00:00 app[web.1]: [Fri Sep 11 2015 19:29:09 GMT-0400 (EDT)] WARNING The HUBOT_AUTH_ADMIN environment variable not set
2015-09-11T23:29:11.595322+00:00 app[web.1]: [Fri Sep 11 2015 19:29:11 GMT-0400 (EDT)] INFO Joining 6795_development@conf.hipchat.com
2015-09-11T23:29:11.611039+00:00 app[web.1]: [Fri Sep 11 2015 19:29:11 GMT-0400 (EDT)] INFO /app/scripts/pugme.coffee is using deprecated documentation syntax
2015-09-11T23:29:11.791665+00:00 app[web.1]: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2015-09-11T23:29:11.791162+00:00 app[web.1]: WARNING: NODE_ENV value of 'production' did not match any deployment config file names.
2015-09-11T23:29:11.607064+00:00 app[web.1]: [Fri Sep 11 2015 19:29:11 GMT-0400 (EDT)] INFO /app/scripts/jenkins-pstat.coffee is using deprecated documentation syntax
2015-09-11T23:29:11.791845+00:00 app[web.1]: WARNING: No configurations found in configuration directory:
2015-09-11T23:29:11.791904+00:00 app[web.1]: WARNING: /app/config
2015-09-11T23:29:11.791956+00:00 app[web.1]: WARNING: See https://www.npmjs.org/package/config for more information.
2015-09-11T23:29:12.141098+00:00 app[web.1]: [Fri Sep 11 2015 19:29:12 GMT-0400 (EDT)] INFO Successfully authenticated to Redis
2015-09-11T23:29:12.174113+00:00 app[web.1]: [Fri Sep 11 2015 19:29:12 GMT-0400 (EDT)] INFO Data for brain retrieved from Redis
2015-09-11T23:31:00.458425+00:00 app[web.1]: VM list pending for 3 more zone(s)
2015-09-11T23:31:00.504942+00:00 app[web.1]: VM list pending for 1 more zone(s)
2015-09-11T23:31:00.557113+00:00 app[web.1]: Determining desired worker distribution across zones
2015-09-11T23:31:00.557148+00:00 app[web.1]: 4 existing workers located
2015-09-11T23:31:00.557229+00:00 app[web.1]: Ignoring VM worker-1442008270-10-zone-b with status: TERMINATED
2015-09-11T23:31:00.557327+00:00 app[web.1]: Ignoring VM worker-1442010334-12-zone-b with status: TERMINATED
2015-09-11T23:31:00.557366+00:00 app[web.1]: Ignoring VM worker-1442002395-zone-a-20 with status: TERMINATED
2015-09-11T23:31:00.557400+00:00 app[web.1]: Ignoring VM worker-1442010334-23-zone-f with status: TERMINATED
2015-09-11T23:31:00.557606+00:00 app[web.1]: No zones have existing workers. Distributing across: a,b,c,f
2015-09-11T23:31:00.558973+00:00 app[web.1]: Placing a max of 7 workers in each zone
2015-09-11T23:31:00.559108+00:00 app[web.1]: Zone a will have workers: 0,1,2,3,4,5,6
2015-09-11T23:31:00.559192+00:00 app[web.1]: Zone b will have workers: 7,8,9,10,11,12,13
2015-09-11T23:31:00.559259+00:00 app[web.1]: Zone c will have workers: 14,15,16,17,18,19,20
2015-09-11T23:31:00.559301+00:00 app[web.1]: Zone f will have workers: 21,22,23,24,25
2015-09-11T23:31:00.559831+00:00 app[web.1]: Launching 7 machines in zone us-central1-a
2015-09-11T23:31:00.566214+00:00 app[web.1]: Creating VM: worker-1442014260-01-zone-a
2015-09-11T23:31:00.579003+00:00 app[web.1]: Creating VM: worker-1442014260-03-zone-a
2015-09-11T23:31:00.588218+00:00 app[web.1]: Creating VM: worker-1442014260-05-zone-a
2015-09-11T23:31:00.596228+00:00 app[web.1]: Launching 7 machines in zone us-central1-b
2015-09-11T23:31:00.600711+00:00 app[web.1]: Creating VM: worker-1442014260-08-zone-b
2015-09-11T23:31:00.609199+00:00 app[web.1]: Creating VM: worker-1442014260-10-zone-b
2015-09-11T23:31:00.617087+00:00 app[web.1]: Creating VM: worker-1442014260-12-zone-b
2015-09-11T23:31:00.627607+00:00 app[web.1]: Launching 7 machines in zone us-central1-c
2015-09-11T23:31:00.632896+00:00 app[web.1]: Creating VM: worker-1442014260-15-zone-c
2015-09-11T23:31:00.642450+00:00 app[web.1]: Creating VM: worker-1442014260-17-zone-c
2015-09-11T23:31:00.650652+00:00 app[web.1]: Creating VM: worker-1442014260-19-zone-c
2015-09-11T23:31:00.662745+00:00 app[web.1]: Launching 5 machines in zone us-central1-f
2015-09-11T23:31:00.668471+00:00 app[web.1]: Creating VM: worker-1442014260-22-zone-f
2015-09-11T23:31:00.680457+00:00 app[web.1]: Creating VM: worker-1442014260-24-zone-f
2015-09-11T23:31:00.465101+00:00 app[web.1]: VM list pending for 2 more zone(s)
2015-09-11T23:31:00.556765+00:00 app[web.1]: VM list retrieved for all 4 zones
2015-09-11T23:31:00.558811+00:00 app[web.1]: Zones not busy:  [ 'a', 'b', 'c', 'f' ]
2015-09-11T23:31:00.560351+00:00 app[web.1]: Creating VM: worker-1442014260-00-zone-a
2015-09-11T23:31:00.572931+00:00 app[web.1]: Creating VM: worker-1442014260-02-zone-a
2015-09-11T23:31:00.583387+00:00 app[web.1]: Creating VM: worker-1442014260-04-zone-a
2015-09-11T23:31:00.593160+00:00 app[web.1]: Creating VM: worker-1442014260-06-zone-a
2015-09-11T23:31:00.596388+00:00 app[web.1]: Creating VM: worker-1442014260-07-zone-b
2015-09-11T23:31:00.605252+00:00 app[web.1]: Creating VM: worker-1442014260-09-zone-b
2015-09-11T23:31:00.613139+00:00 app[web.1]: Creating VM: worker-1442014260-11-zone-b
2015-09-11T23:31:00.620793+00:00 app[web.1]: Creating VM: worker-1442014260-13-zone-b
2015-09-11T23:31:00.627904+00:00 app[web.1]: Creating VM: worker-1442014260-14-zone-c
2015-09-11T23:31:00.636820+00:00 app[web.1]: Creating VM: worker-1442014260-16-zone-c
2015-09-11T23:31:00.646745+00:00 app[web.1]: Creating VM: worker-1442014260-18-zone-c
2015-09-11T23:31:00.656489+00:00 app[web.1]: Creating VM: worker-1442014260-20-zone-c
2015-09-11T23:31:00.662836+00:00 app[web.1]: Creating VM: worker-1442014260-21-zone-f
2015-09-11T23:31:00.674455+00:00 app[web.1]: Creating VM: worker-1442014260-23-zone-f
2015-09-11T23:31:00.686512+00:00 app[web.1]: Creating VM: worker-1442014260-25-zone-f
2015-09-11T23:31:02.207331+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-12-zone-b with operation-1442014261085-51f811bcb0349-3c257d1d-fef0c5d4
2015-09-11T23:31:02.214392+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-16-zone-c with operation-1442014261078-51f811bcae7f1-ec38bb9c-21fa63a8
2015-09-11T23:31:02.217451+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-15-zone-c with operation-1442014261078-51f811bcae7f0-46e77fa9-71ad061c
2015-09-11T23:31:02.238148+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-10-zone-b with operation-1442014261080-51f811bcaefc0-76b7e762-1001bab0
2015-09-11T23:31:02.252770+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-02-zone-a with operation-1442014261086-51f811bcb0731-6f49668e-83379ac4
2015-09-11T23:31:02.261019+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-06-zone-a with operation-1442014261090-51f811bcb16d0-24ecb2c5-de60b0f5
2015-09-11T23:31:02.269702+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-19-zone-c with operation-1442014261088-51f811bcb0f01-9eacb423-e2fac9d0
2015-09-11T23:31:02.271623+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-04-zone-a with operation-1442014261112-51f811bcb6cc1-6772b547-ca375aeb
2015-09-11T23:31:02.304956+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-21-zone-f with operation-1442014261088-51f811bcb0f01-e635da6a-ef37a1db
2015-09-11T23:31:02.307531+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-09-zone-b with operation-1442014261087-51f811bcb0b1a-449464cd-17bf3dc7
2015-09-11T23:31:02.307771+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-05-zone-a with operation-1442014261081-51f811bcaf3a9-abb8c68a-ae9c67b2
2015-09-11T23:31:02.308811+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-03-zone-a with operation-1442014261078-51f811bcae7f1-59fbf8ab-ecd21895
2015-09-11T23:31:02.345514+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-11-zone-b with operation-1442014261086-51f811bcb0731-6bec4432-38552ccf
2015-09-11T23:31:02.363719+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-07-zone-b with operation-1442014261079-51f811bcaebd9-82a78839-f11ec9d9
2015-09-11T23:31:02.211415+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-14-zone-c with operation-1442014261095-51f811bcb2a59-35bdd694-624baac3
2015-09-11T23:31:02.215487+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-20-zone-c with operation-1442014261090-51f811bcb16d1-d2e21175-abd1b03b
2015-09-11T23:31:02.231989+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-17-zone-c with operation-1442014261081-51f811bcaf3a8-9f13d1e4-16a05327
2015-09-11T23:31:02.251205+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-13-zone-b with operation-1442014261081-51f811bcaf3a9-fb6c7431-16903d5e
2015-09-11T23:31:02.255237+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-18-zone-c with operation-1442014261080-51f811bcaefc0-16c6d45e-d6bf6f1b
2015-09-11T23:31:02.262586+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-01-zone-a with operation-1442014261080-51f811bcaefc3-3f6dea26-6cb88894
2015-09-11T23:31:02.270550+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-00-zone-a with operation-1442014261029-51f811bca2889-0b962de2-25e2907a
2015-09-11T23:31:02.304745+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-08-zone-b with operation-1442014261078-51f811bcae7f1-6fa05d22-7943b36d
2015-09-11T23:31:02.307345+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-23-zone-f with operation-1442014261082-51f811bcaf791-2dbb301b-0da41b34
2015-09-11T23:31:02.309519+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-24-zone-f with operation-1442014261097-51f811bcb3228-61211c1e-afb62ab4
2015-09-11T23:31:02.348498+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-25-zone-f with operation-1442014261089-51f811bcb12e8-ed594228-5399e8ef
2015-09-11T23:31:03.474683+00:00 app[web.1]: VM creation call succeeded for: worker-1442014260-22-zone-f with operation-1442014261142-51f811bcbe1f2-b6b77858-c44d5350

[jlward]

Looks good :+1: