Service name from K8s breaks encapsulation and is inconsistent with universal mode

jamesdbloom commented 3 years ago

Summary

Problem

Currently the service name in K8s is generated as follows:

fmt.Sprintf("%s_%s_svc_%d", svc.Name, svc.Namespace, svcPort.Port)

I believe this breaks encapsulation as clients need to know which namespace the pod is deployed in for that K8s cluster. A service can not be deployed to both K8s and onto VMs (i.e. universal mode) because the service names won't match. A service also can't be deployed to different namespaces in different K8s clusters because the service names won't match.

It doesn't seem correct that clients need to know which namespace a service is deployed into as this is a leaky abstraction.

This means there is no way to fail over from K8s to EC2 or between K8s with different namespaces.

I understand kubernetes namespaces are used for domain names for services internal to a K8s cluster, but I don't think copying this pattern to something which is exposed outside a cluster makes sense. Kubernetes Ingress Controller is a better analogy where service name can be completely decoupled from the namespace and details of how and where a service is deployed.

Proposed Solution

Ideally the default name could be overridden by an annotation on the K8s pod / service. If the service was exposing multiple ports there are several strategies that could be adopted:

expose first port only if default name is overridden
- this would mean to expose multiple ports multiple services would be needed
allow each port to be separated overridden
ignore the annotation if multiple ports are exposed
- this would mean to expose multiple ports multiple services would be needed
if multiple ports are exposes appended the port to the service name (but not if only a single port was exposed)

lahabana commented 3 years ago

Such a feature makes sense. I'd go with overriding with an annotation We have this type of annotation for port: 80.service.kuma.io/protocol: http.

So you'd have something like: 80.service.kuma.io/service: myService. WDYT?

jamesdbloom commented 3 years ago

@lahabana I think that sounds like a good option.

jakubdyszkiewicz commented 3 years ago

Hey James,

I hear your arguments and I totally agree with them. I'd love to fix this problem, but there is a caveat here - security.

Let's say we allow 80.service.kuma.io/service: myService. Assuming there is a service redis in namespace team-a. Nothing stops me from creating malicious service in another namespace team-b named redis and collect the traffic that is intended to go to the original redis.

On Universal this problem is solved because we are generating dataplane token for kuma.io/service: redis.

Correct me if I'm wrong but so far I saw that the security scope of Kubernetes is bound to a namespace. What I mean by that is that a team member has usually access to the team's namespace, that's why service name is also bound to a namespace.

We could introduce service name override but we also need to have a thing that restricts users to use it. Otherwise, it defeats the mTLS. We have the same problem on Kubernetes with labels (there is no mechanism to restrict users how to place them) and also Mesh.

One of the options that come to my mind is Service/Labels/Mesh Affinity which would be Kubernetes only objects. For example, if I know that team-a can use Service redis, as a Mesh operator I could place this

kind: ServiceAffinity
metadata:
  name: redis-affinity
spec:
  affinities:
  - service: redis
    namespaces: ["team-a"]

Given your experience, would something like this be a viable option?

jamesdbloom commented 3 years ago

@jakubdyszkiewicz you make a good point about security.

The ServiceAffinity could work but I suspect it also would be an extra level of complexity and coordination between mesh operators and teams. So although it technical works and would meet the security requirements I'm not sure its a good option.

I guess what would help is support for RBAC and namespaces in kuma. Similar to what you can do in kubernetes but applied to service configuration and service deployment in kuma across both K8s and universal mode. This would also help to reduce coordination between mesh operators and teams, because teams could be self-service.

So perhaps this should be parked until RBAC and namespaces could be supported.

jakubdyszkiewicz commented 3 years ago

I guess what would help is support for RBAC and namespaces in kuma

Could you expand on namespace here? Did you mean to somehow support Kubernetes namespace in Kuma? Or to have a concept of Kuma Namespace so that Mesh can be split into a couple of Kuma Namespaces but at the same time one Kuma Namespace can gather many Kubernetes Namespace?

jamesdbloom commented 3 years ago

The coupling between a kubernetes namespace and a kuma namespace isn't something I'd thought about too much but I agree it could potentially cause issues. The basic idea I was thinking about in relation to RBAC and namespaces was as follows:

RBAC to control service configuration, service deployment and zones joining the mesh

i. RBAC for Configuration

a namespace / context could be a sensible way to model the set of configurations / services managed by a team (i.e. role)
allows full access (i.e. read and write) for configuration (i.e. Traffic Permissions, Traffice Routing), of services owned by a team (i.e. within a single namespace)
limits / prevents access for teams (i.e. read-only or no access) for configuration of services not owned by a team
only allows write access to mesh operator teams for specific policies (i.e. Traffic Metrics, Traffic Log) so these can be centrally managed and are only read-only for non-operator teams
only allows write access to mesh operator teams for default policies (i.e. Retry, Timeout, RateLimit, CircuitBreaker) so defaults policies can be centrally managed but can be overridden by non-operator teams for their services configurations
ii. Identity Federation for Configuration
identity federation & role / access mapping, such as via OIDC Federation
allows RBAC to be implemented for configuration changes
for example:
- plugin interface in kumactl that allows a mesh operator to develop a plugin for retrieving token from federated identity provider, appropriate for their organisation
- the plugin would enable a custom authentication journey i.e. via weblink displayed in terminal or a non-web prompted steps in terminal, such as: enter username, enter password, choose 2FA, etc
- kumactl to send token to CP to authorise configuration changes based on JWT claims and role / attribute mapping
  iii. RBAC & Identity Federation for Service Deployment
RBAC for service deployment to control which services are allowed to be deployed to which namespaces
allows service ownership to be proven i.e. which services can be deployed into a namespace:
- allow service owned by a team to be deployed into the teams's namespace
- prevent services not owned by a team to be deployed into the team's namespace (to avoid a rouge service stealing traffic)
for example:
- tags / annotations in service deployment (i.e. universal or K8s) include token (i.e. JWT)
- service deployment sent to CP to authorise the creation of dataplane token for a specific namespace
  iv. RBAC for Namespaces in a Zone
RBAC for controlling which namespace is allowed for a given zone
ensures K8s administrator can't deploy corrupted zone CP that removes security
global CP should only distribute namespaces information from zones that are allowed to deploy a namespace
v. RBAC & Identity Federation for Zones
RBAC / identity federation to create zone join tokens
improves flexibility and self-service support around teams creating zones in their own K8s clusters
ensure teams can easily create zones that can deploy services for their namespaces

Does that make sense?

lahabana commented 2 years ago

This is related to: https://github.com/kumahq/kuma/issues/3238

github-actions[bot] commented 2 years ago

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

github-actions[bot] commented 2 years ago

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

jamesdbloom commented 2 years ago

Is there any plan to address this issue?

github-actions[bot] commented 1 year ago