search

uselagoon / remote-controller

A group of controllers for handling Lagoon builds and tasks in Kubernetes or Openshift
5 stars 1 forks source link

Detect available docker-hosts and distribute builds across them #239

Open tobybellwood opened 10 months ago

tobybellwood commented 10 months ago

In a scenario where there are N docker-hosts in N zones, the remote-controller should be able to provide a build pod with the connection details for it's closest docker-host. The build will then use that docker-host for the duration of it's build.

shreddedbacon commented 7 months ago

Since we know that the topology aware hints stuff doesn't really work, we can reject that. I don't think we will have an easy way to properly do topologically closest docker-hosts.

Alternatively though, we could have the remote-controller scan for kind: Service within a cluster that have a specific label (eg app.kubernetes.io/component=lagoon-remote-docker-host) to determine if there are 1 or more docker-hosts within a cluster. Or we can define custom labels to use to select docker-hosts too.

When the controller is creating the build pods, it can just round robin them amongst the known docker-hosts by injecting the docker-host record into a build, rather than the builds defaulting to docker-host.lagoon.svc, that can be a fallback.

tobybellwood commented 7 months ago

I still think it may be worth trying to identify the "closest" zone-based (per labels) docker-host and allocating that to a build in this scenario, but round-robining if necessary (overload or unavailablility)

shreddedbacon commented 7 months ago

I still think it may be worth trying to identify the "closest" zone-based (per labels) docker-host and allocating that to a build in this scenario, but round-robining if necessary (overload or unavailablility)

The main problem though, is that the controller doesn't schedule the builds, and there is no easy way for a build pod to identify which zone it is in to know which to choose from, but agree, if it is possible to solve this without the remote-controller having to handle scheduling too, then that would be nice.

shreddedbacon commented 2 weeks ago

Re scoping this to be able to inspect the docker host(s) deployed in a cluster and try to distribute builds amongst the available docker hosts.