Allow to specify HAproxy image in cluster configuration

[harmjanblok]

What would you like to be added: An option in the cluster configuration to specify a non-default HAproxy image.

Why is this needed: In a lot of corporate environments only proxies or proxy-like systems, for example JFrog artifactory, can be used to pull docker images. The kindest/node image can be specified in the node part of the cluster configuration, when running HA clusters, pulling the kindest/haproxy fails due to missing internet connectivity. Being able to override the HAproxy image would allow to specify a different HAproxy image to work with.

[BenTheElder]

I think the main problem here is, HA is an underdesigned / under-utilized feature and the configuration is coupled to a known haproxy version.

It should be possible to configure the container runtime + registry to provide a mirror of kindest/ in the meantime, FWIW.

I'm very curious to hear the multi-control-plane use-case as well, this functionality was intended for testing Kubernetes itself, it's not expected to make sense for other purposes ... 👀

[aojea]

Being able to override the HAproxy image would allow to specify a different HAproxy image to work with.

but you can always download the haproxy image with docker-save and restore it in the local docker store where kind is running, can't you?

[harmjanblok]

but you can always download the haproxy image with docker-save and restore it in the local docker store where kind is running, can't you?

@aojea yes my current workaround was to download the image and relabel it, so it was present on my local machine.

[harmjanblok]

I'm very curious to hear the multi-control-plane use-case as well, this functionality was intended for testing Kubernetes itself, it's not expected to make sense for other purposes ... 👀

@BenTheElder We use an application which uses Kubernetes as 'backend storage' for its application state, it turns out that this setup is fragile as it is latency-sensitive. I wanted to use kind to simulate latency increases, for example on etcd leader changes. Although I can simulate etcd leader changes, the latency is far below the production latency, probably due to various other factors, like load on the API server. So I wouldn't claim this as a real multi-control-plane use-case.

[BenTheElder]

Thanks for sharing!

for example on etcd leader changes

FWIW the current multi-control-plane mode does not even have multiple etcd (only one on the first control plane) as kubeadm does not provide this out of the box and "actual HA" in this way is not something we're testing even for kubernetes at the moment.

I would probably suggest simulating the latency changes manipulating the network, @aojea has some samples for this I believe.

Also yes, there's multiple workarounds for airgapped access to the haproxy image, but I'm sure it's not ideal. I'm not sure we're prepared to handle arbitrary images though.

Perhaps instead of specifically configuring this image we could support telling kind to rewrite kindest/ => some user specified mirror for all images (and in the future the same instead for whatever our default registry is if it changes as well) by way of an environment variable? This would also allow configs to more broadly remain portable across the airgap. (alternatively it should already be possible to configure the host container runtime to do this, without any kind changes)

[aojea]

I would probably suggest simulating the latency changes manipulating the network, @aojea has some samples for this I believe.

https://gist.github.com/aojea/603e88ba709aac874bd7611752261772

demo here https://kccnceu2021.sched.com/event/iE3g/kubernetes-advanced-networking-testing-with-kind-antonio-ojea-redhat

[harmjanblok]

@aojea thanks, I tested with tc qdisc by increasing and decreasing the delay, hoping for a reproducible setup, but didn't work out as expected, so I used a 'real' cluster for further investigations.

@BenTheElder yes I think the airgapped haproxy is only used for a very limited number of use-cases. For the kindest images we currently use the image setting for a node in the Cluster configuration, which works fine until now.

[jpmcb]

Big +1 to this.

Over at tanzu-community-edition, we have a lite-weight deployment model of our kubernetes platform that can be deployed ontop of kind which also allows users to specify multiple nodes (by using the kind configuration).

I'm very curious to hear the multi-control-plane use-case as well

This gives our users the ability to quickly deploy and test the platform in a multi-node setting on their local workstations ontop of kind without having to deploy the whole app platform to the cloud (because 💸 and 🕐 )

We have heard feedback from users that since kindest/haproxy is pulling from docker and not our internally hosted image repository, we hit rate limiting in this multi-node scenario when using a CI/CD pipeline that has to pull alot of images.

[BenTheElder]

FWIW there are multiple workaround options for this and I'm open to fleshed out proposals. I made a rough suggestion above but there hasn't been feedback on this approach yet.