Implement agent reuse, toggled by a cluster profile config option

[brandonvin]

Description

This implements agent reuse following the approach outlined in these comments: https://github.com/gocd/kubernetes-elastic-agents/issues/53#issuecomment-1399398498 https://github.com/gocd/kubernetes-elastic-agents/issues/53#issuecomment-1406967391. Resolves https://github.com/gocd/kubernetes-elastic-agents/issues/53

Cluster profile has an option to enable agent reuse, defaulting to false:

The naming, and presentation of this option, are open to feedback! 🙂

Changes

When pods are created, they are annotated with a hash of the elastic config. This ensures that when elastic config is changed, agents created from the old config will not be reused, and will eventually expire after the timeout.

When agent reuse is enabled, the main behavior changes are:

• Job completion request -> instead of terminating the agent, mark the agent as idle so it may be considered for reuse
• Should assign work request -> assign work only if the agent is available for work and the elastic config hash matches the agent
• Create agent request -> only create an agent if none are available for reuse

Other supporting changes:

• KubernetesInstance is an immutable data class constructed by a builder, and no longer contains a Kubernetes client.
• Improving field names (e.g. properties) for clarity between cluster profile properties and elastic profile properties.
• Refreshing instances moved inside the ServerPingRequestExecutor instead of "inline" in the KubernetesPlugin request handling block.
• Added a request handler for REQUEST_PLUGIN_SETTINGS that returns an empty map - just to silence some warnings about this request not being implemented.
• Some classes are refactored to allow testing with less mocking.
• Removed an unnecessary semaphore.

This ended up being a pretty large set of changes. Happy to explain in more detail if needed!

Testing

Many unit tests updated and added, and run with ./gradlew test. I've also tested this running GoCD on my machine with a Kubernetes cluster in kind.

[chadlwilson]

Thanks for this. Appreciate all the work here. Unfortunately I have limited capacity to review, try out and give useful insight so might take me some time 🙏

[woopstar]

Nice work +1 on this

[brandonvin]

Hi there @chadlwilson! Is there any chance that this PR could move forward in the next month or so? Is there anything I can do to help, or make you more comfortable merging this? If not - no worries - I would probably take a stab at forking the repo, building from a fork, and testing this out more in our production GoCD server. If I find any issues from that I'll push them up to this PR.

[chadlwilson]

Thanks for the ping.

Honestly the challenge with these elastic agent plugins is just my concerns around regression and most don't have any particular "integration" tests when actually working with GoCD - and there are a lot of weird timing and other scenarios that can arise with the elastic agent plugins. (e.g what happens if the GoCD server dies, when it comes back does it still correctly adopt the agents or do they end up orphaned? what happens if a pod dies that was expecting to be reused? what if the container restarts inside?). This plugin also happens to probably be the most actively used elastic agent approach.

But ya, the real limitation has been time and energy, distraction and the extra 'friction' on validating this one. it is still on my radar, I've just been kicking the can down the road. But it's a bit easier for me to validate locally now that we have smaller non-Alpine arm64 container images to use on elastic agents.

I naively expected you were probably running this off your own build anyway, is that not the case? :-)