EAP Kubernetes Cloud Target Discovery doesn't support execution containers for health checks

[geofflamrock]

Team

• [X] I've assigned a team label to this issue

Severity

No reports yet of this blocking customers

Version

2022.2.6249+, any 2022.3 version

Latest Version

No response

What happened?

When a Kubernetes cluster is discovered during a deployment using cloud target discovery, a health check is performed which requires kubectl to exist on the path. Some workers do not have kubectl installed on them, including dynamic Ubuntu workers in Octopus Cloud and we recommend that an execution container is used on these workers. Kubernetes targets support setting a health check container image on them which supports this use of execution containers.

The issue arises when discovering a Kubernetes target via cloud target discovery as we don't support setting a container on a discovered target, so the health check fails on these workers without kubectl.

Reproduction

• Configure a deployment process with a Kubernetes deployment step to run on a worker that doesn't have kubectl and configure the use of an execution container. In Octopus Cloud Ubuntu dynamic workers do not have kubectl.
• Configure cloud target discovery by tagging the Kubernetes cluster and setting credentials in project variables
• Perform a deployment

Target discovery will be performed and the cluster discovered, but the health check will fail as kubectl does not exist in the path and the execution container is not used.

Error and Stacktrace

During the health check on the discovered target: Could not find kubectl. Make sure kubectl is on the PATH.

More Information

Example of the error occurring:

Workaround

Workaround is to use a worker that has kubectl installed on it.

[twerthi]

@geofflamrock I've just run into this on our Samples instance when I was configuring an example to show customers. How do you implement the workaround? Once the target was discovered and added, I encountered the could not find kubectl error. Within Infrastructure, I changed the details so it used a different worker pool for the health check, however, when the deployment executed, it set the worker pool back to what it was. Is there something else that needs to be done?

[geofflamrock]

@twerthi

Once the target was discovered and added, I encountered the could not find kubectl error. Within Infrastructure, I changed the details so it used a different worker pool for the health check, however, when the deployment executed, it set the worker pool back to what it was.

This is the way that the cloud target discovery works, which is a bit unfortunate for the workaround here, on the second deployment the target associated with the cluster gets rediscovered and updated, which reverts the worker pool back to the one used during discovery.

A couple of other things you might be be able to try here:

• The worker pool you updated the target to use for the health check, are you able to use this on the deployment step as well? If so that should then pass through to the target for the health check.
• If you are discovering an AWS EKS cluster, we have an additional variable Octopus.Aws.WorkerPool you can use to specify the worker pool that should be used during discovery, which should then also pass through to the target. At the moment we don't have an equivalent variable for Azure unfortunately.

[twerthi]

@geofflamrock

When updating the dynamically added target, I switched from running on the worker itself to using Execution Containers which allowed the health check to continue. All of the workers that are on Samples don't have have kubectl explicitly installed, so we rely on the worker-tools image. Is that an enhancement that might be able to be made? Supply an additional project variable Octopus.Worker.ExecutionContainer.Image or something like that?