search

kubernetes-sigs / gateway-api

Repository for the next iteration of composite service (e.g. Ingress) and load balancing APIs.
https://gateway-api.sigs.k8s.io
Apache License 2.0
1.77k stars 460 forks source link

GEP: Support namespace scoped implementations #567

Open howardjohn opened 3 years ago

howardjohn commented 3 years ago

What would you like to be added:

Currently, someone deploying an implementation must have cluster scoped privileges to access GatewayClass. Often in multitenant clusters, teams are given access only to a set of namespaces without permissions to access cluster scoped resources. Users in this scenario should be able to deploy an implementation of the API.

For example, they would deploy an nginx proxy controller, a LoadBalancer Service, a Gateway, and some routes. This is really close to working, but they still need to create some GatewayClass.

Potential options

  1. Cluster admin creates generic GatewayClass with no params, say "in-cluster-proxy". NS admin creates a Gateway referencing it. They still need to configure parameters of the gateway (ie things that would typically exist in parametersRef of Gatewayclass), which they cannot do in GatewayClass. As a result, they need to externalize these configurations to somewhere else (annotations on the deployment/service/gateway, nginx-specific configmap, etc). This isn't great since now its entirely implementation specific how things are configured - but its also not too bad, as parametersRef is all implementation specific anyways. Still requires some cluster admin coordination

  2. Same as (1), but maybe they don't even bother with a generic GatewayClass and just put some bogus value there (it is a required field)

  3. Do not support deployments with only namespace permissions

howardjohn commented 3 years ago

cc @costinm

robscott commented 3 years ago

We discussed this at the last community meeting. It sounded like there was significant interest in deployment models without the GatewayClass resource. Maybe something that could look like this:

  1. Each Gateway implementation can choose to support a set of static, domain-prefixed classes (such as gke.io/xlb). These would not require GatewayClass resources to exist.
  2. Users could specify those classes in the same way they specified any other class on a Gateway resource, there just wouldn't be a GatewayClass resource.
  3. GatewayClass resources matching these static names would be ignored by the implementation with the exception of potentially adding some kind of status indicating the conflict with the static class.

/cc @danehans @hbagdi @bowei

SantoDE commented 3 years ago

While I agree with you @robscott (for the need of namespace scoped installations), I feel the same issue is similiar to the normal IngressClass Ressource, no? As this is also a cluster scoped ressource. So maybe its possible to find a solution that works for both sides?

re: https://github.com/kubernetes/kubernetes/issues/99824

I feel like, we should try to solve that all together? 🤷‍♂️

howardjohn commented 3 years ago

Each Gateway implementation can choose to support a set of static, domain-prefixed classes (such as gke.io/xlb)

We had mentioned before the GatewayClass is optional, but if its added it will be used (to use parameters in the GWC). If we use this syntax I don't think its possible since you cannot have / in the name? I guess your third comment is conflicting with that, so maybe that was just my impression that wasn't shared by others. I think its useful to be able to overwrite it.

robscott commented 3 years ago

Thanks @SantoDE for linking this to the upstream issue! I agree that whatever we should try to gain consensus in the broader sig-network community as well to ensure any change can help with Ingress as well as Gateway.

You're completely right @howardjohn, the example I gave of gke.io/xlb wouldn't work. My goal was to define some patter that would make conflicts unlikely or impossible. For example, it would be bad if 2 controllers both claimed the xlb class if there was no formal GatewayClass definition. Although we could use some kind of subdomain format, that seems rather unnatural. Maybe the format should simply be {implementation-name}-{variation}. So to follow up with my example above, gke-xlb. That seems rather unlikely to conflict with anything, but there are others that may be trickier. IE there is more than one nginx ingress implementation.

I think its useful to be able to overwrite it.

I'm not completely tied to the idea that GatewayClasses of these names would be invalid/ignored, but I still prefer it. If implementations were to provide standard classes with internally consistent meanings, it could get confusing if that class name meant something slightly different in another cluster. Maybe more significantly, if a GatewayClass resource could be used to override the meaning of one of these static class names, every controller would have to be able to read from the cluster-scoped GatewayClass API, which would be a significant hurdle for namespace-scoped deployments.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

jpeach commented 3 years ago

/remove-lifecycle stale /lifecyce frozen

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

hbagdi commented 3 years ago

/lifecycle frozen

robscott commented 2 years ago

I think our best solution here would be to provide implementations with a pattern they can follow if they want to support single-namespace deployments. For example, implementations could provide a flag to indicate that they should only watch resources within the namespace they are deployed along with a flag to indicate which the class name they should implement. This class name should not overlap with any cluster-wide class names to avoid multiple implementations trying to implement the same resource.

akutz commented 2 years ago

I think our best solution here would be to provide implementations with a pattern they can follow if they want to support single-namespace deployments. For example, implementations could provide a flag to indicate that they should only watch resources within the namespace they are deployed along with a flag to indicate which the class name they should implement. This class name should not overlap with any cluster-wide class names to avoid multiple implementations trying to implement the same resource.

Maybe we should revisit https://github.com/kubernetes-sigs/gateway-api/issues/136? It still strikes me as the most elegant solution. It is not about managing infra, but about pointing to infra within the multi-tenancy world in which we live on Kubernetes. Without a proper ClusterGatewayClass and GatewayClass, I argue everything else is a work-around to the most correct solution.

Regarding your proposal @robscott, I'm not sure how that is diffrent than what GWC used to have back in early days?

https://github.com/kubernetes-sigs/gateway-api/blob/fd5ffd1db31fb9a5b6dff1010679bcf42777738d/apis/v1alpha1/gatewayclass_types.go#L67-L84

It kind of feels like this has come full circle...

youngnick commented 2 years ago

There are a few problems I can see with introducing a ClusterGatewayClass and associated GatewayClass namespace-scoped resource:

I guess I'm saying that I don't agree that ClusterGatewayClass is the more correct option, and changing to it at this point would be an api-breaking change that would require v1beta2. I don't think providing the functionality in the top comment of the issue is worth the months of work it would take to do a v1beta2. We would need to do a full pass through v1alpha3 or something, then move to v1beta2.

To put this another way, I don't think there's a way to make compatible changes to the API to get us to a NamespacedGatewayClass at this point, and I don't know if this is a big enough issue for the whole community to spend the effort to roll the apiVersion. (I acknowledge that's not very satisfying for people who need a resolution).

I think that the workarounds proposed about not requiring an actual GatewayClass object to be present are probably the best we're going to be able to do without a new apiVersion.

shaneutt commented 1 year ago

@howardjohn does this represent a known customer need for Istio (or any other implementations you work on) or is it more theoretical? If this is more of a theoretical need given the difficulties I feel slightly inclined to close it in favor of waiting for a time where the needs of end-users can move it forward. In either case do we feel comfortable with saying we wouldn't need this for GA, as we should be able to use the ObjectName reference in the current GatewaySpec pretty open-endedly? Let me know your thoughts :thinking:

howardjohn commented 1 year ago

The main thing is params need to be namespace scoped. Having GatewayClass is not too bad if you just have 1 for the entire cluster, but per-namespace owners need to be able to deploy and configure Gateways. This is handled by https://github.com/kubernetes-sigs/gateway-api/pull/1757 though.

https://istio.io/latest/docs/setup/additional-setup/gateway/#dedicated-application-gateway is an example in Istio

arkodg commented 1 year ago

@shaneutt this is a very common use case, based on teams, tenancy, and ownership of apps/components within a single K8s cluster environment, there is a need to support a wide range of cases where

Implementations can solve this by adding a notion of deployment mode to decide which namespaces to watch resources in and which namespaces to deploy managed infra in, but since GatewayClass is cluster scoped / not namespace scoped, we have to trust users (without any formal restriction) creating the GatewayClass to link it to the right implementation controller, (rather than the intent and capability of the controller itself to make this call)

shaneutt commented 1 year ago

Good, makes sense to me but seeing active support for this gives me extra confidence in prioritizing it sooner rather than later so let's move this out of triage then and consider it something we'd like to solve for v1.0.0.

shaneutt commented 1 year ago

Definitely still wanted by many implements and is a common use case, but doesn't need to hold up GA: we should be able to add this functionality in a post-GA world without too much hassle.

shaneutt commented 6 months ago

It seems like this still has obvious benefits to the ecosystem, despite the lack of anyone to champion it and drive it forward. Let's remove the frozen lifecycle from it at least so it can remind us that it's hanging out back here, and we can continue to re-evaluate.

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

costinm commented 3 months ago

One option for this (and perhaps few mesh cases) is to just make GatewayClass optional. It effectively is optional for GAMMA - there is no requirement to have a GatewayClass if parentRef of HttpRoute is Service, and clearly not the end of the world.

A Gateway object may still need to have a parentRef - we may want to relax this, but even if it remains required - I don't think we reject the creation of Gateway if GatewayClass is missing ( it may be installed later, eventual consistency - or not available in that cluster). An implementation that is namespace scoped will simply match on parentRef - even if there is no actual object. Not worse than mesh, where we don't even have something to match on.

Worst case - such an implementation would not pass the compatibility, which probably will be the case since most cross-namespace features will also not work. But in practical terms - it will solve the user problem.

Would be nice to have a pattern - where the parentRef is allowed to be namespace/name, with name pointing to some Service for example, like we did for GAMMA - the Service would be the one associated with the per-namespace gateway.

This can go a bit further: from 'namespace' gateway to 'workload gateway' ( also sometimes called 'sidecar'). A workload gateway would run as a container next to some other workloads - the parentRef and configs would use some equivalent of the workload selector - perhaps a Service that selects the workloads.

Nothing discussed here require real changes to the spec - just a bit of tolerance with implementations that may not fully support things that can't be supported in specific cases.

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

snorwin commented 2 months ago

/remove-lifecycle rotten

shaneutt commented 2 months ago

Hi @snorwin :wave:

I see you've bumped the lifecycle on this, is this something you're interested in working on personally?

snorwin commented 2 months ago

@shaneutt, thanks for reaching out.

I'm currently facing similar challenges with implementing the Gateway API for a security gateway managed by an operator that supports different installation modes (e.g., AllNamespaces, Multi-Namespace, OwnNamespace). The primary reason for these different modes is to enable multi-tenancy. While I'm aware that multi-tenancy can be achieved using multiple gateway classes, it seems that cluster-scoped resources are often considered red flags in shared or managed k8s clusters.

I appreciate @costinm's suggestion of "headless" gateways and believe this issue should move in that direction. For this reason, I have removed the rotten lifecycle label. I'm also happy to contribute to this topic.

shaneutt commented 2 months ago

Cool, thanks for the insights! We can definitely provide support if you'd like to try and help move this one forward, maybe a good next step would be to add something to the agenda for one of our upcoming community meetings so we can just generally discuss the topic, please feel free to add something there and let us know what you're thinking!

youngnick commented 2 months ago

@snorwin, if you could drop the use cases you're interested in here, then we can talk some more about this.

To reiterate what I said in the community meeting today - my main argument here is this:

Controllers that cross multiple namespaces (which most Gateway API controllers do), necessarily run at cluster scope, because there is no API construct that is a larger scope than a single namespace, but smaller scope than cluster scope.

So, every workaround that we do to make it so that you don't need to create a cluster-scoped API construct is building in confusion - because you now have a larger-than-namespace-scoped thing (the set of namespaces the controller needs to watch) that has no representation in the API (because there is no place to store a set of namespaces that's outside a single namespace).

As it stands, if we use a workaround like a specific name for a GatewayClass or similar, all that means is that the state of Gateway API objects is no longer entirely described by API objects. There's some extra information that you need to have that you cannot retrieve from the Kubernetes API itself (for example, you're running in GKE and have a standard set of GKE classes available to you using magic strings).

The requirement that a system of Gateway API resources can be entirely understood by looking only at Kubernetes API objects is a bit implicit, I guess, but it's always been there for me - it's a reaction to the Ingress domain-name-claiming problem where a namespaced object (an Ingress) claims a cluster scoped resource ( a hostname ). That has really bad effects, like being able to steal another services' traffic if you are a bad actor. I really want to avoid that sort of thing happening for Gateway API.

None of this is to say that I don't think we should continue talking about this, I just think that we need to be really careful that we're not creating a silent problem that will mess us up later.

snorwin commented 2 months ago

@youngnick thank you for your thoughts and input. I will try to illustrate the use case I have in mind in more detail:

Ana, an application developer, wants to use the Gateway API to manage the connectivity of her micro-service application. She works in the financial and insurance services industry, which still operates most of its infrastructure in on-premise data centers (private cloud). In this environment, Kubernetes is often provided as large shared clusters (also known as Namespace as a Service) managed by an external provider, where Ian and Cherio work. The Gateway API is already installed on the cluster, but Ana wants to deploy her own Gateway infrastructure self-contained within her application namespace. This is due to the lack of flexibility and the operational overhead of coordinating every change with Ian and Cherio.

Self-contained, as mentioned above, means that all the Gateway API resources (e.g., Gateway, HTTPRoute), as well as the controller running with namespaced RBAC (i.e., Roles and RoleBindings), and the resources generated, are all within a single namespace.

As highlighted in the community meeting, my preferred solution would be to make the .spec.gatewayClassName optional - why? This explicitly indicates that the Gateway has no GatewayClass and is managed within a namespaced scope. In contrast, using gatewayClassName: none offers no guarantee that a GatewayClass named none won't be created in the future, potentially allowing another controller to take over this Gateway. To prevent clashes between different Gateway controllers within a namespace, e.g., leader election could be used. A Gateway without a GatewayClass would neither solve nor worsen the domain delegation issue because, as far as I know, most controllers don't guarantee the uniqueness of hostnames across all Gateways within a GatewayClass.

So far, I have only described the self-contained namespace use case. While true multi-namespace tenancy doesn't exist, the illusion of it can be created using namespaced or cluster-scoped approaches, such as in combination with admission policies. I agree that it is not feasible to define a clear solution for multi-namespace approaches using the Gateway API, and it is also unnecessary to provide a guidance for an anti-pattern.

shaneutt commented 2 months ago

I'm uncomfortable with removing specification in order to suite this need :thinking:

That said, what we really need right now is for someone to be assigned to this issue, and to start a GEP: organizing some brainstorming sessions as needed, and otherwise writing up all the options discussed so far, and any new ones we can come up with so we can discuss the pros and cons of various approaches. If anyone wants to be assigned to this and is ready to push it forward, please say so and we'll be happy to support you in your efforts.