Scheduling applications to a k8s namespace based on CF space

[zankich]

As far as I can tell eirini only supports specifying a single namespace for running user applications: https://github.com/cloudfoundry-incubator/eirini#configuring-opi-with-cloud-foundry-and-kubernetes

Is it possible to configure eirini to schedule applications to many namespaces? I'd like to be able to deploy applications bound to a CF space to a dedicated k8s namespace.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172154654

The labels on this github issue will be updated when the story is started.

[julz]

Hi @zankich - as of today we assume just one namespace for all CF apps, but only because we haven't heard strong use cases for doing better yet and one namespace was therefore The Simplest Thing That Could Work.

We'd be very receptive to ideas about better ways to do this, but we'd need to think through what the best boundary is: for example should a CF space == namespace or should it be CF org == namespace or should it be CF isolation segment == namespace?

Can you outline a bit more about your use case for wanting a specific space, and do you have strong opinions about whether it should be at the org, space or isolation segment level?

[cwlbraa]

To support tenancy constructs between space developers, I'm pretty confident the mapping should be space == namespace. If we map things at a higher level, there will be no way we can ever hope to offer kubectl access to developers as they'll be able to access their space-neighbor's resources. Namespaces are the most granular tenancy construct available in k8s, and spaces are the most granular available in CF. It's easier to construct less-granular tenancy groupings than it is to divide what is already the most granular thing available.

They both have the word "space" in there, too! Very convenient.

In terms of use cases, there are lots once we assume a world where k8s is the only backend we care about. Space quota enforcement could be pushed down to namespaces. Data services could be provisioned and shared in new ways without exposing them to entire orgs or foundations.

There's even some potential that we could externalize CF's Authorization model into k8s, effectively a sort of perm_ng using k8s Users, Roles, and Rolebindings.

That said, I'm not entirely sure that OPI needs to concern itself with the CF<->k8s mapping for spaces. Each request could take an additional, optional namespace parameter and error if that namespace isn't already available.

[julz]

I think that makes a lot of sense @cwlbraa, I've created https://www.pivotaltracker.com/story/show/172350759 for the Eirini side.

Very cool that this way the problem of figuring out the right mapping of CAPI concepts to namespaces can live in CAPI, which seems obviously right (though I guess this means we've going to leak the concept of "namespace" out of OPI, I think that's fine tho -- we really don't need to pretend at this point that we can productively avoid leaking kubernetes concepts around).

[loewenstein]

@julz It is probably hijacking, but as isolation segments were mentioned above I thought I'd add my 2 cents anyway.

I would like to see Eirini deploying CF apps not only into separate namespaces, but also into separate k8s clusters. I suppose the mapping would be isolation segment == cluster. Would it make sense, to introduce such a capability early on together with the introduction of a namespace parameter?

[loewenstein]

@zankich Is there already an issue or tracker item for the CAPI side of this? I would comment that we might want to go for cf-<org>-<space> as a naming pattern to prevent clashes with non-cf namespaces and between different cf orgs.

But just mentioned here it will go lost, as Eirini should stick to simply using whatever namespace was provided.

[cwlbraa]

@loewenstein We're also interested in multicluster scheduling, but getting working CF apps in multiple clusters would require a bunch of common system components deployed and configured in those clusters -- Would scheduling just the LRPs across clusters have some benefit that I'm missing? I'm going to go make a CAPI issue.

[loewenstein]

A working CF app would need an envoy as Istio ingress gateway and another as sidecar. In addition, logs and metrics would have to be scraped. What other system components am I missing?

We would gain an isolation level between apps similar to that of cf4bosh with isolation segments or at least that is what I suppose. Currently, with cf-for-k8s we do not even get this isolation between apps and system components.

[cmoulliard]

for example should a CF space == namespace

If possible, I will be very interested to have this level of relation between a space and namespace as by example RBAC or resources to be assigned could be defined at this level

[cmoulliard]

CF org

We could use the org name/level to tag the applications, using the k8s label, part of that org as org: my CF org in order to let a user to query easily all the applications, k8s resources belonging to that org. Of course, this proposition could also be extended to the space level to also create a k8s label for the space - space: my space

[cmoulliard]

perm_ng using k8s Users, Roles, and Rolebindings.

If you can manage that correctly, that would be fantastic as currently there is no easy way to manage RBAC, assign it to a user's profile (developer, ...) under kubernetes