search

concourse / hush-house

Concourse k8s-based environment
https://hush-house.pivotal.io
30 stars 23 forks source link

hh: enforce hard antiaffinity on web nodes #17

Closed cirocosta closed 5 years ago

cirocosta commented 5 years ago

Hey,

Having jobs that stream in and out a lot, we might end up having problems with the web pods if those end up being scheduled to run in the same machine as the whole network bandwidth of the machine might be consumed by them, while in reality, those two could be serving the full bandwidth of two VMs.

For instance, consider the following case:

screen shot 2019-03-03 at 9 09 48 pm

There we have two web nodes in the same VM consuming the whole 1Gib of network bandwidth that the instance has (2 250Mbit TX + 2250MBit RX).

If those workloads were split in two instances, we could go even higher.

cirocosta commented 5 years ago

It's there! https://github.com/concourse/hush-house/blob/dff9e7e0ac2661c94a44d916bf28422dd7f1adb3/deployments/with-creds/hush-house/values.yaml#L36-L48

Funny thing: the throttling there was actually at the disk of the receiver that would slow down the whole streaming of the contents.