[FEATURE]: kubernetes-container Deploy action

[ShankyJS]

Feature Request

Background / Motivation

The container type started as a basic implementation that could serve multiple providers instead of tailoring it exclusively to Kubernetes.

As time passed, Garden users continued using this module as it's simple and doesn't require too much management, providing a cleaner syntax with easier adaptability. However, we have reached a point where we can no longer extend the container type, leaving users with open features requests pending.

This is why, as part of this discussion, we are requesting the creation of a kubernetes-container type that expands the simple container one.

What should the user be able to do?

Users should be able to use this new kubernetes-container type but have access to more in-deep customization of different resources, creating a production ready type that is war-tested, has lots of built-in capabilities, and should be extensible.

Why do they want to do this? What problem does it solve?

By removing the complexity of Helm, or Kubernetes manifests, users could only focus on building their kubernetes-container type once and then concentrate exclusively on coding their applications.

Also, this new type could have features like horizontalPodAutoscaler, VPAs, NetworkPolicies, and other specific K8s resources that could make this type even more powerful.

Suggested Implementation(s)

I'm not sure how we are templating our resources into Kubernetes, but I would build a powerful Helm Chart (Like Helmet), for example, with everything a user might need to deploy.

• Deployment
• Service
• Ingress
• PVC/PV
• HPA
• VPA
• NetworkPolicies
• ConfigMaps
• Secrets
• Etc.

Example of a strong Helm Chart that offers multiple features.

And then template it into Garden with the details the user will provide.

How important is this feature for you/your team?

🌹 It's a nice to have, but nice things are nice 🙂

[chulkilee]

What would be added to / removed from k8s manifest attributes when using garden's kubernetes-container? Probably templating, dependencies / some controls (e.g. disable)?

I'm wondering how it would remove the K8s manifest complexity too.

[eysi09]

This is a hot topic and something we've been discussing for awhile so I want to add my 2 cents here. Also updated the title to use action instead of module

I personally think that we should NOT add a kubernetes-container Deploy action. It's a strong opinion but fairly loosely held.

Here are my reasons in no particular order:

• Now we have yet another way to do similar things that need to be documented and explained and we'll constantly need to answer the question: When should I use container vs kubernetes-container vs kubernetes Deploy action.
• The container Deploy action is a good and simple starting point. Once you find yourself needing things like nodeSelector or initialDelaySeconds, doesn't that suggest you just need the full-blown kubernetes Deploy action? As opposed to some in-between action. We can have tools for generating the config from the container Deploy action.
• The kubernetes-container Deploy action will not work with other systems as is, e.g. CD tools like ArgoCD. However, if you're using the kubernetes Deploy action you by definition have K8s manifests that work transparently with other tools.
• Similarly, you're likely already running your stack on K8s in production which means you already have said manifests and should probably re-purpose those with Garden via the kubernetes action, as opposed to duplicating config and managing some of it with Garden and the rest not.
• You can use action templates to keep things DRY and manageable. In this case you can define a template for a container Build action and a corresponding kubernetes Deploy action once, and re-use in multiple places.
• You can also use Garden's templating to re-use manifests. The files spec for, say, an API Deploy action could look like this: files: [./manifests/base/service.yml, . /manifests/base/deployment.yaml, ./manifests/api/ingress.yml] to make a simple example. Since you can use Garden template strings in K8s manifests, you can e.g. override names, labels and such in the base manifests. (Note that using Garden template strings in manifests means they won't work with systems like noted above, but Garden also supports Helm and Kustomize.)
• Our motto is "automation over abstraction", and adding a kubernetes-container action is not that, it's an abstraction that will leak.

So instead, I'd suggest:

• Documenting these common patterns a lot better
• Adding functionality to generate config for a kubernetes Deploy action from a container, e.g. via the plan command we have on the roadmap.

[porterchris]

@eysi09 I'm curious how you've used/re-used action templates, pulled in shared manifest files, etc. Do you have an example project or could you put one together and push it up to show how you've handled the kubernetes type? We've found that the jump from container to kubernetes is quite far and sometimes frustrating.

[eysi09]

Yeah for sure, I think it's best I just go ahead and add a couple of example projects to the repo showing this pattern.

[alexkuretz]

I'd like to see those examples as well, as I'm one of the requesters of this feature that lead to this issue being created.

• The container Deploy action is a good and simple starting point. Once you find yourself needing things like nodeSelector or initialDelaySeconds, doesn't that suggest you just need the full-blown kubernetes Deploy action? As opposed to some in-between action. We can have tools for generating the config from the container Deploy action.

This position is rather arbitrary, is it not? Why include livenessTimeoutSeconds but not initialDelaySeconds? Why is there a PVC deploy action but no PV deploy action? My suspicion is that someone on the dev team either needed or didn't need these features at the time they were created rather than it being specifically planned as the "right amount" of automation over abstraction. I do understand the reluctance to add every/many options to the container action.

For some context, we've got 10 projects running with Garden and have been able to get away using container modules (and now starting to move to actions) in every case except for the PV I mention above where I had to write a manifest for a kubernetes deploy action. I need initialDelaySeconds and had originally asked for nodeSelector but would actually prefer topologySpreadConstraints as we're now using Karpenter for autoscaling. I definitely don't relish having to move everything to kubernetes actions just to get control over where the pod ends up. EDIT: Looks like I might be needing serviceAccountName too.

[eysi09]

Thanks for the added context @alexkuretz!

And I hear you, having to migrate is definitely one of the downsides to my suggestion above. I'd be curious to hear if the examples I'm working on make that more straightforward.

However, this position is definitely not arbitrary and I'll try to explain why below.

The idea with the container module/action is for it to be "abstract" and work with multiple providers.

In a given environment, you might deploy your container to a Kubernetes cluster, in another you might deploy it as a serverless function, e.g. to AWS Lambda or Google Cloud Run.

So we've been very intentional about not having anything K8s specific leak into what's suppose to be a general purpose container "deployment". (Maybe with some exceptions, but usually such that the field in question could conceivably apply to other platforms and is generic enough).

With actions, this separation is even clearer in my opinion. There's a container Build action and a platform/tool specific Deploy action (e.g. kubernetes or helm but could also be, say, aws-lamda) that references the build.

That's why we'd rather add a third action type, the kubernetes-container Deploy action, which is kind of like container but exposes more K8s specific fields. At the same time, we'd probably want to deprecate the container Deploy action (but keep the Build action) since in its current state it's basically legacy behaviour from the "module days".

Or, as I've been arguing for, not add another action type but rather encourage users to use the kubernetes Deploy action and ensure it's easy to migrate and maintain.

[eysi09]

If we take the kubernetes-container route, we could potentially make it fairly easy by:

1. Start treating the current container Deploy action as a kubernetes-container. Basically add kubernetes-container as an alias and deprecate the container type name. Assuming this isn't a breaking change.
2. Start adding K8s specific fields to the action spec, again treating the container Deploy type as a kubernetes-container (with a legacy name)
3. Add support for other runtimes separately. So for e.g. running containers with Docker, we could add a docker-container type.

So we're not adding a new type, just re-purposing an existing one under a new name (also supporting the old one). But again assuming it doesn't break anything and that there's currently no way to use a container Deploy action outside of K8s (I'd need to double check).

It's still an open question how we handle the abstraction part. Do we just add more and more fields as people request them? Or simply allow people to overlay manifests like Kustomize?

It's these challenges that make me partial to just using the kubernetes Deploy type.

[worldofgeese]

I believe this is mostly a problem of documentation. Teams have the tools, today, to produce abstractions consumable by their devs that support these requested fields without us needing to create new action types or extending existing types to support.

To add to the options @eysi09 gave: we support Helm, and by extension, Helm Library Charts. Library charts like The Helmet are plug-and-play. nodeSelector is supported, initialDelaySeconds is supported, topologySpreadConstraints is supported, serviceAccountName is supported. These are just key values to The Helmet and can be set directly inside your garden.yml file.

For an example of how these are set, have a look at my own personal project. We also use the helm Deploy type in our getting started tutorial for Golang users created by @ShankyJS, where we're using cookiecutter to render whole action blocks out.

@alexkuretz you'd only need to bring your container action then add in a helm Deploy to have access to every value you've requested. The helm action is just for setting your values for any field. No writing your own Kubernetes manifests. I also wrote a blog post that includes 0-to-deploy instructions under the heading, “Helm: the package manager for Kubernetes”.

I think we can do a lot better getting folks comfortable using Helm because most teams don't want to be writing their own Go templating to support their needs. With library charts, they don't need to.

[alexkuretz]

We've not gone the way of writing our own helm charts, but I can see The Helmet being useful, that's cool how the library chart works. I will say I wouldn't have thought to go this route as especially when the garden docs state this :

Note: If you only need a way to deploy some Kubernetes manifests and don't need all the features of Helm, you can use the simpler kubernetes action type instead. Check out the kubernetes guide for more info.

Our perception when we initially started using Garden was that a big part of the value was in the abstraction of k8s resources since it can be a ton of load for a new dev to understand manifests. We've definitely relied heavily upon Garden's features in this regard, it's only as our projects have evolved that we've started hitting some of these limitations.

Given that context we're still fans of a kubernetes-container deploy action.

[eysi09]

As discussed, here's an example of using the kubernetes Deploy action but having a shared set of manifests that only need to be written once and developers mostly don't need to worry about: https://github.com/garden-io/garden/pull/4992

~I'll also follow up with an example that uses action templates.~ EDIT: We already have that example here and the corresponding docs here. We still need to ensure these docs are visible to those getting started with the K8s plugin.

At this point, we have a bunch of ways to mostly achieve the same thing, so following this we'll write a doc on best practices and when to pick what.

That's very much a thing we've been discovering on our end as well as we see Garden being used in the real world :)

I know I've steered slightly off topic here and this feature request is still valid, but at least we have some documented workarounds.

[alexkuretz]

Thanks for the example and docs link, we took a brief look at them, the example is quite sparse but your PR is much more helpful. It appears the ConfigTemplates are a way for us to build our own custom kubernetes-container deploy action rather than easily leverage one provided by Garden. While we definitely appreciate having that flexibility, it seems like a lot of up-front work on our part for something that to my understanding from previous discussions with Garden team members has been requested from multiple other customers as well.

[worldofgeese]

@alexkuretz moving to a helm action wouldn't take much more than the better part of an hour with the examples I have provided. I'm happy to connect with your team to demonstrate how to migrate your annotations and values. We'll continue to put in effort into the docs to showcase these capabilities and how easy it is to make effective use of these actions that can scale with your needs. Config Templates do require a bit more upfront work but they're very powerful in the hands of a Platform Team or equivalent. A Helm Library Chart powering the helm action requires a single init and transfer of values. To create a brand new action means supporting values on-demand from our community and customers for limited UX returns.

Very happy to schedule a session any time next week with you and your team.

[eysi09]

the example is quite sparse but your PR is much more helpful

I agree, I'm re-working that using the same project as in the original PR + adding more docs.