search

kubernetes-sigs / gateway-api

Repository for the next iteration of composite service (e.g. Ingress) and load balancing APIs.
https://gateway-api.sigs.k8s.io
Apache License 2.0
1.84k stars 480 forks source link

Feature request: suggest to provide a helm chart for gateway-api and related CRDs #1590

Open wilsonwu opened 1 year ago

wilsonwu commented 1 year ago

What would you like to be added: I'd like to have a helm chart to easily install and upgrade gateway-api.

Why this is needed: A usage standard, I can contribute it.

youngnick commented 1 year ago

Seems reasonable, I guess. This should also optionally install the validating webhook?

wilsonwu commented 1 year ago

Yes, I think so, this requirement is coming from that I am working on a Contour based Microservices API Gateway, we need to keep the k8s cluster with gateway-api installed, and I think it is a better way to make community to provide an official helm chart and maintained it by community.

shaneutt commented 1 year ago

I'm hesitant, but open to discussing. If there are some clear and strong motivators to do this I think it could be worth discussing at some length in our next community meeting.

robscott commented 1 year ago

I'm not sure a Helm chart is a good fit for Gateway API. Here's what Helm's docs have to say about CRDs:

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations

I understand and appreciate their hesitancy to support updating or deleting CRDs and I think more broadly that highlights how careful we need to be when installing and updating CRDs. I'm hoping that the role of CRD management will largely be handled by the cluster provider in the future. For example, on GKE, Gateway API CRDs can be installed and upgraded automatically for you: https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#enable-gateway.

wilsonwu commented 1 year ago

I'm not sure a Helm chart is a good fit for Gateway API. Here's what Helm's docs have to say about CRDs:

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations

I understand and appreciate their hesitancy to support updating or deleting CRDs and I think more broadly that highlights how careful we need to be when installing and updating CRDs. I'm hoping that the role of CRD management will largely be handled by the cluster provider in the future. For example, on GKE, Gateway API CRDs can be installed and upgraded automatically for you: https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#enable-gateway.

Yes, this is helm chart design that can't upgrade and delete CRDs, In our practice, we manually manage CRDs in private K8s cluster, maybe we can request some feedback from end users for this.

youngnick commented 1 year ago

I didn't even think about the CRD install issue, I think that makes this probably a -1 from me.

tomkerkhove commented 1 year ago

Eventhough Helm does not support it properly, in reality a lot of/majority of OSS tools do manage them through Helm though.

rainest commented 1 year ago

Helm's CRD management system means that you need to manage them via another tool after the initial install, so I don't see much that a chart would provide beyond a small reduction in install friction via a dependency (absent first party support for kustomize URL dependencies in Helm itself).

In practice, from what I've seen in the wild, I'd expect many clusters to have a CD system (Argo or similar) to ensure kustomize URL dependencies are deployed.

wilsonwu commented 1 year ago

As per our comments, close it, when we have more requirement, let's reopen.

lucasfcnunes commented 1 year ago

@wilsonwu, could this issue be reopened?

wilsonwu commented 1 year ago

@wilsonwu, could this issue be reopened?

Yes, reopened.

youngnick commented 1 year ago

I don't think anything has changed though - I don't see us creating a Helm chart for Gateway API CRDs any time soon unless something has changed about Helm and CRDs I haven't heard of?

lucasfcnunes commented 1 year ago

@youngnick As @tomkerkhove said:

Eventhough Helm does not support it properly, in reality a lot of/majority of OSS tools do manage them through Helm though.

I think this comment is more relevant than taken. Prometheus CRDs, Grafana CRDs, Cert-manager CRDs, Cilium CRDs, Elastic CRDs, Istio CRDs, Velero CRDs, Argo CRDs, etc all use charts for deployment diffing and cleaning (it's hard to know when to remove something unneeded).

youngnick commented 1 year ago

The additional problem that we have on top of the usual CRD management problems is that it's possible (maybe even likely) that multiple implementations that require the CRDs will be installed in the same cluster, and they would then need different versions of the Helm chart (if we provided one).

I worry that we could end up with implementations having their CRDs downgraded while they're running, because another implementation was installed in the cluster.

This is different to all the examples @lucasfcnunes mentioned in that all of those CRDs are owned by only a single implementation.

I'm not saying that we shouldn't do something to make this experience better, just that I don't think the solution can be "make a Helm chart" without addressing some of the concerns that arise from being a shared set of CRDs rather than a set of CRDs closely tied to a single implementation.

lucasfcnunes commented 1 year ago

I worry that we could end up with implementations having their CRDs downgraded while they're running, because another implementation was installed in the cluster.

We just don't use /crds folder https://github.com/helm/helm/issues/10585 and go with https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-2-separate-charts instead of https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-1-let-helm-do-it-for-you

Helm would actually protect us from downgrading (installing over current resources) as each resource is tied to a single release. Helm never takes over when the resources don't match the release's annotations and labels. A simple kubectl apply doesn't do that.

jimangel commented 11 months ago

I'm +1 for a community owned helm chart install for the CRDs and I want to pitch another item that would be nice to include:

Ability to generate Gateway API related objects (HTTPRoutes, TLSRoutes, etc)

Why?

To me it seems like there's a couple options:

{{- if .Values.ingress.enabled -}}
{{- $fullName := include "demo.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
  {{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
  {{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
  {{- end }}
{{- end }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
...

Today, I'm working around this limitation with: https://github.com/jimangel/unofficial-gateway-api-helm

That does not include the CRDs, but in my opinion, I think the project could have some generic templating of Gateway API spec objects AND a /crd folder in the helm chart. Thus, allowing the end user to use the helm built-in: --skip-crds flag (https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) if they don't want to use the CRDs.

I think the trade-off would be documentation around upgrades / life-cycle. The alternative is more custom YAML and more custom solutions like I've already started. It would be great to have a centralized solution for the Gateway API spec to increase adoption.

The helm chart could be positioned as a secondary option, as in, "Gateway API project suggests installing via kustomize..." and then below have the helm options for advanced users and/or spec templating.

shaneutt commented 11 months ago

Just for my own personal clarity on where people are coming from here and what's actually being suggested, I wanna know what people's expected commitment level is here:

Is anyone in this thread specifically interested in owning and maintaining this long term? Or is this more of a request to have the current project as it stands take on the responsibility?

robscott commented 11 months ago

I still have the same concerns about CRD management directly via Helm, but will be the first to admit that Gateway API CRD management is nowhere near as easy as it could/should be. I've created https://github.com/kubernetes-sigs/gateway-api/issues/2678 to discuss an alternative installation method that could also be bundled in a Helm chart.

jimangel commented 11 months ago

Is anyone in this thread specifically interested in owning and maintaining this long term?

@shaneutt I already have the burden of this with https://github.com/jimangel/unofficial-gateway-api-helm; and without a community chart, I would imagine we'll have more fragmented solutions as everyone makes their own "wedge" to solve this problem vs. a community standard. I would gladly help maintain and support a solution, assuming there's one we all agree on.

I kind of thread-jacked a bit with my rant on objects beyond CRDs, and honestly, I think that's going to be an increasingly bigger problem. I would love to see other OSS helm charts adopt support for Gateway API objects, similar to how we see with ingress today, but there's not even published guidance or standards around that.

I still have the same concerns about CRD management directly via Helm

These concerns are valid. As we chatted about, it's not an issue for external secrets operator or cert manager. But, to be fair, those are the more than only CRDs.

Given the "out-of-tree" spec for Gateway API, there needs to be a way to unify it with an implementation AND there needs to be a way to unify it with your workloads. Today, there is no standard and it's causing a lot of friction (especially for anyone trying to automate their clusters at scale using GitOps + OSS helm charts/kustomize).

I would be open to the discussion in #2678 for an alternative to install CRDs (defining the "official" way for other implementations/charts to include the Gateway API). Then my remaining gripe for "how can we make gateway api included in all the helm charts as a standard, like ingress" could be solved per workload.

If we define what that "official" way is, I'm happy to help write the docs and guidance so others can adopt this model.

It's less about helm and more about the fragmented state of things today and the friction caused by Gateway API being out of tree with no defined guidance on how other helm charts / OSS workloads can integrate seamlessly with BYOgateway API integration.

shaneutt commented 8 months ago

This has been stale for some time, if someone wants to champion this please let us know as that's what it will take to move it forward.

networkhermit commented 7 months ago

This has been stale for some time, if someone wants to champion this please let us know as that's what it will take to move it forward.

@shaneutt Hello. I just create a poc of crds-only helm chart and it works. See the draft pr #2951 for more detail. If the general direction is alright, I can finish the remaining todos and get the pr merged.

Edited:

The draft pr is on hold and awaiting further discussion. Sorry I didn't read through all the context, i.e. the complex requirements we want it to achieve.

See https://github.com/envoyproxy/gateway/issues/1721#issuecomment-2035558087 for an example to incorperate crds installation in iac/gitops.

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

jimangel commented 6 months ago

👋 hey all, checking in because I've been MIA on paternity leave. I'll be back in office mid-June, happy to help see this to completion or help @networkhermit when I return.

/remove-lifecycle rotten

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

youngnick commented 3 months ago

I can see folks are still interested here, so I'll unstale this for now.

I'm also adding this as "help-wanted", but this will definitely require some close work with myself and other maintainers, so it's definitely not a good first issue. Solving this properly will require more work on recommendations for CRD version management, which is a very poorly defined area at the moment.

/help-wanted /remove-lifecycle stale

k8s-triage-robot commented 4 days ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale