search

concourse / hush-house

Concourse k8s-based environment
https://hush-house.pivotal.io
29 stars 23 forks source link

hh: experiment isolating Strabo final jobs #58

Open cirocosta opened 5 years ago

cirocosta commented 5 years ago

Hey,

As a result of the exploration in https://github.com/concourse/hush-house/issues/22, I'm fairly confident that if we move the final two jobs from strabo to a dedicated worker that has no more containers than just this, we'll be able to gain huge performance benefits for them, while not impacting the workloads that by chance (we use random placement strategy here) end up in the worker where that task runs.

To do so, we could have a very small pool of workers (a single worker would be sufficient) in which we could run a single tagged Concourse worker that would be the target used by strabo.

E.g.:

NODE POOLS

    generic-1 (2 nodes, each 4CPU, 16GB ram)
        concourse web
        prometheus server
        jaeger ...

    workers-1 (5 nodes, each 16CPU 32GB ram)
        hush-house's default pool of beefy workers

    strabo-1 (1 node, 4CPU, 16GB ram)
        hush-house's strabo workers 

HELM DEPLOYMENTS

    hush-house
        (concourse web)

    worker
        (concourse worker)

    strabo
        (concourse worker --tag=strabo)

STRABO PIPELINE

    the very same, except that the two tasks that aggregate
    all inputs are tagged with `--tag=strabo` (all gets remain
    the same)

This way, the strabo-1 node would never have more than ~2 containers, leading us to pretty much just userspace CPU utilization, and not that aberration that we see at the moment :grin:

Thanks