Add K8s native solution for running the agent

glena commented 3 months ago

Hello!

Vote on this issue by adding a 👍 reaction
If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

We have a naive example on how to run agents in K8s (https://github.com/pulumi/customer-managed-deployment-agent/pull/3).

Some customers want proper k8s support using native k8s resources (ie: having an operator dispatching jobs)

Affected area/feature

EronWright commented 3 months ago

Seems like it would be pretty straightforward to write a Kubernetes operator that acted similarly to the self-hosted agent, dispatching deployments as Kubernetes Jobs.

An argument could be made for or against doing the work in the Pulumi Kubernetes Operator (as opposed to a new operator). A possible benefit would be a unified backend for running deployments driven by Pulumi Cloud or by the Stack CRD.

I would advocate for a layered design based on a low-level CRD, similar to JobDefinition, that would cause the operator to spawn a Kubernetes Job to execute the defined job. The status block would reflect the status of the step(s). This becomes a building block to be used in two ways:

for the self-hosted agent use case, via a higher-level CRD, e.g. Agent, that takes jobs from the Pulumi Cloud.
for the classic Pulumi Kubernetes Operator use cases, via the Stack CRD.

cleverguy25 commented 1 month ago

I am not sure why we would need a CRD here for this, rather than just a deployment that is capable of creating Jobs.

pulumi / customer-managed-deployment-agent