search

pantsbuild / rbe-token-server

Pants server to generate remote build execution tokens.
0 stars 4 forks source link

Increase size of worker pool when we issue a token; decrease it when it expires #13

Open illicitonion opened 5 years ago

illicitonion commented 5 years ago

https://cloud.google.com/remote-build-execution/docs/api/reference/rest/v1alpha/projects.instances.workerpools/patch allows us to change the size of a worker pool.

Right now, our utilisation isn't great. During weekends and night times, we're paying for 16 idle workers. During peak development times, we're queueing actions.

Instead, let's default to 1 worker, and whenever we issue a token bump by 16 workers, and an hour later (or better yet, if we get an RPC from the travis shard earlier than that, but with the 1-hour fallback still as a safeguard) decrease by 16.

We should probably still have some maximum number of workers, just in case, or alert if our worker-hours get over some threshold, or something.

wisechengyi commented 5 years ago

The link is 404, probably permission issue.

Also if 16 is the status quo, then we'd probably need 2-3x during business hours.

illicitonion commented 5 years ago

Yeah, that link will only work if you're whitelisted in our GCP project. @stuhood can add folks.

stuhood commented 5 years ago

We should do this as the next effort in the area of test remoting. In the meantime, I'll increase our fixed worker count.