Research item: Multiple function namespaces for a single OpenFaaS installation

[alexellis]

Description

Multiple function namespaces for a single OpenFaaS installation #433

Expected Behaviour

The behaviour would allow functions to be deployed to N different namespaces, perhaps one per user or team.

Current Behaviour

Either one installation per team or user or use a different mechanism for grouping functions such as prefixes and annotations/labels as in OpenFaaS Cloud.

Possible Solution

The solution would need additional RBAC permissions - potentially to even create new namespaces in the cluster making faas-netes effectively a cluster admin. This is unlikely to be a viable configuration for restrictive environments.

I'm open to suggestions on implementation that are pragmatic when it comes to more restrictive environments.

[matipan]

I like the idea of having this feature. From an operator point of view having to maintain a single OpenFaaS installation and still provide a more isolated environment for different teams/users is better than keeping up with multiple installations.

How would this look like from a developer point of view? Will I have to specify the namespace on the yaml of the function, a label maybe?

I'm not sure about how to implement this on a more pragmatic way, my knowledge of Kubernetes management is limited. But I will look into alternatives and let you know what I find.

As a side note: this also aligns well with OpenFaaS Cloud. We will not only be isolating the routes of each user, but also separating where those functions live.

[alexellis]

As a starting-point we could start with statically provisioned SAs/RBAC and the ability to deploy to different namespaces, then after that look at how to automate it for other downstream projects like OpenFaaS Cloud.

You would extend the helm chart with a new parameter like:

--set namespaces=openfaas-teama,openfaas-teamb,openfaas-teamc

For user deployments perhaps using something like an annotation would be sufficient:

functions:
  teamafunction:
    image: image1:latest
    annotation: group=openfaas-teama
  teambfunction:
    image: image2:latest
    annotation: group=openfaas-teamb

[alexellis]

Matching 1:1 between namespace and stack.yml annotation name may not be ideal for larger systems because we can only support a group of "dev" for the first user that takes it.

An alternative could be to use labels on the namespace to lookup a namespace:

kubectl create ns generated-value-here
kubectl label ns/generated-value-here group=teama

[LucasRoesler]

I really like the idea of using labels on namespaces, but we need to ensure that we handle the case when labels arent unique. Even if it is accidental, surfacing the issue clearly will be important.

[alexellis]

I guess we might also have issues similar to those found with the use of labels before - it's a DNS-only name which is allowed, probably not much more permissive than the plain-old namespace name.

[LucasRoesler]

What if we allow the admin to specify a file with a map of function annotations to namespaces? We could then use a config map to provide this map as via file and build faas-netes to watch if the file changes. This would still allow dynamic configuration and allow the use of any annotation on the function.

When the file is loaded, we can validate it and log the errors to stdout and we could also track a metric of config errors, so that admins could alert on configuration errors, if they wanted.

The current function namespace flag could be used as a default, when a function has no annotation.

In theory, we could even allow a combination of annotations to be specified for a single namespace, e.g. group=A,env=prod: prodNamespaceA (this might be a "nice to have" feature)

We might consider adding a flag that requires all functions to have a specific annotation or set of annotations.

I could even imagine a version of this for the openfaas-operator that is specified via a CRD, similar to how ingress are handled by the nginx-controller to construct a nginx config file.

[alexellis]

If we focus on how the controller would create deployments, there is a question I have.

For faas-netes we don't use the watch API, so we have no additional cost of imperatively doing CRUD over N namespaces, even 100.

With the Operator we would need all our watches duplicated per namespace - i.e. Pod/Deployment/Service x N namespaces, or would we be able to watch globally and filter down?

[stefanprodan]

We can use a global watch and filter the events based on a list of namespaces but that means the operator will need a ClusterRole. I think filtering is the easies way of doing this.

[alexellis]

Lister / Informer creation / binding:

https://github.com/openfaas-incubator/openfaas-operator/blob/master/pkg/controller/controller.go#L88

Sync for specific namespace:

https://github.com/openfaas-incubator/openfaas-operator/blob/master/pkg/controller/controller.go#L304

[alexellis]

From experience talking to OpenShift & enterprise users, I could imagine that a ClusterRole may be out of the question for some companies. What other options are there?

[stefanprodan]

Another option would be to deploy one operator per namespace

[alexellis]

In that model I guess it would look a lot like multiple OpenFaaS installations, but we could configure it to share Prometheus/NATS to reduce the footprint.

We need to find out how much of a deal-breaker ClusterRole is for companies.

[LucasRoesler]

To avoid a ClusterRole, couldn't we use a single operator with multiple RoleBindings to access multiple namespaces. The admin could then choose to give openfaas access to the whole cluster or only a subset of namespaces. In the second case, we simply need to make sure we surface the error in a nice way to the user, when the function can not be deployed because a namespace does not exist or is not accessible.

[alexellis]

That would cover the RBAC, but how would the Lister/Informers work? https://github.com/openfaas/faas-netes/issues/433#issuecomment-493657326

[LucasRoesler]

It is possible to create the informers with a namespace, e.g. https://godoc.org/k8s.io/client-go/informers#SharedInformerOption

But it isn't clear that it is needed because we don't currently restrict the namespace of the informer.

The listener does set a namespace, but this is done on the fly inside syncHandler, but I think @stefanprodan can answer better about the listeners.

[alexellis]

What if we had Prometheus/NATS and etc in a core/system namespace, and then one operator with a finite scope and Role per namespace? Does that work?

[paurosello]

If the OpenFaaS is meant to be a platform service, I would be inclined to give it a ClusterRole as an operator but from OpenFaaS point of view, I think it should not matter.

In the end, if we have the ability to deploy a function in a specific namespace the operator will decide if they want to give a Role or a ClusterRole to the OpenFaaS installation and this is the functionality that I will try to contribute to the chart.

So in my opinion the chart might offer three options:

• Role in a single namespace (the one with all other OpenFaaS pieces)
• ClusterRole that will allow it to deploy in all namespaces
• Specifying a ServiceAccount that will allow you to customize to your needs

[alexellis]

Here's me invoking functions in different namespaces

Changes I made:

• Used the Operator
• Patched Role/RoleBinding to ClusterRole/ClusterRoleBinding (we could just make this a helm chart option today, there is no harm in it)
• Set functionNamespace to "" and removed the code that sets a default to openfaas-fn if "" is given in the Operator's server.go
• Patched gateway to allow A-Za-z90-9. <- adding . so that /function/fn.ns would be allowable

Other limitations:

• Had to turn off scale from zero because we can't get a function status without knowing its namespace, the name is not enough on its own
• Set direct_functions_suffix to "" so that the URL could pass the namespace / suffix i.e. .default in /function/figlet.default

[alexellis]

I have drafted a diagram which I also shared on Slack in the #kubernetes channel. There are some comments from the community in the channel which will be lost due to limits of the free tier of Slack.

Some people have expressed that they would like to see a new URL scheme to access functions in a non-default namespace such as /functions/namespace/function-name/. A new URL scheme would be a breaking change and would mean deep code changes across every component in the OpenFaaS stack.

The /function/name/ convention is encoded in: all documentation, tutorials, workshops, event-connectors, private integrations, the CLI, in SDKs and more.

I think the change should focus on the smallest possible change domain, but I am open to adding additional vanity URLs if those requesting them are prepared to estimate, carry out and maintain the changes on an ongoing basis.

Some concerns I've heard are:

• We don't want to have a namespace in a URL

As per my diagram, you can use FunctionIngress to create a pretty-URL to whatever domain you like for each public function

• We want to see a tenant name in the URL instead of the .namespace/ notation

This could be covered by an additional vanity URL, however it isn't practical to make huge breaking changes to every component in the stack to work on a new URL scheme.

There is a difference between the way internal components interact with the system and how you serve content to your users.

Here is where I want to set the scope of the work. This issue is about enabling the multi-namespace use-case, rather than building a multi-tenant system.

Multiple namespaces can be used for: projects, environments (staging/e2e/prod), teams, integration testing, and to some extent, tenants.

Multiple tenants requires more than a URL scheme, it requires changes for authz deep into the stack. For the work on multi-tenancy see OpenFaaS Cloud comparison grid.

[pchico83]

I am a strong believer in remote development, where each developer has access to its personal namespace. This way, developers can share cluster services like logs aggregators, metrics, service mesh... the OpenFaaS installation would a great example of this scenario. As a cluster administrator, I would like to install OpenFaas once and provide OpenFaaS credentials for each developer. Depending on the authenticated user, OpenFaaS will deploy the developer function to the corresponding developer namespace.

[alexellis]

@pchico83 I think this is out of scope for the current PR, which is more of a technical discussion on accessing multiple namespaces. OpenFaaS Cloud holds the multi-user story that you're referring to and we've chosen to replace the API with git - i.e. "CRUD operations" on functions is simply: git push

[doowb]

I have a question about how the functions in different namespaces are intended to be deployed. Is it possible to either change the name through the cli when deploying from the stack.yml or add a flag to the cli to specify the namespace the function should be deployed to?

My main concern is if this is intended to be used for staging deployments, I don't want to edit my stack.yml to promote my staging deployment to production. Right now, I can do this using multiple OpenFaaS installs and use the --gateway flag to deploy to the environment I'm targeting.

To implement that, I would propose adding a --namespace flag to faas-cli, as well as, allowing namespace to be specified in stack.yml so the function name doesn't have to be changed. The gateway code would then deploy the function to gateway.openfaas:8080/function/{name}.{namespace} if that's the chosen URI to keep breaking changes minimal.

The other option is to use multiple stack.yml files for each environment and specify the one to use with the -f flag. This is easier and will get an example implementation finished sooner.

Also, in Slack, I liked the idea of having a URI scheme as gateway.openfaas:8080/{namespace}/function/{name} but your argument to the breaking changes required has changed my mind for this case. I think using external routing mechanisms is a better course.

[alexellis]

I've opened https://github.com/openfaas/faas-netes/issues/511 to track the work required and its progress.

[rajibmitra]

Tested this, works fine. @alexellis

[alexellis]

In order to support use-cases like the event-connectors, which must traverse all functions to know which expose a topic, and the UI where users may want to get an overview of what is deployed, or the CLI where users need the same, I would suggest the following:

• Add an endpoint to list namespaces - the UI/CLI would consume this, users still create them manually
• Add a label to openfaas namespaces such that we don't show what is in kube-system, etc

Then "list all" becomes 1 + N API calls:

• API call to namespaces
• N API calls to list functions within each namespace

Even if we don't want to have this kind of a chatty interface, it has to happen somewhere, given the way the Kubernetes clientset works.

An alternative is that "list all" becomes the default mode of the /system/functions/ endpoint, which can then be further refined with ?namespace=X

@johnmccabe @stefanprodan @LucasRoesler I'd like your input on this please.

[martindekov]

I don't mind having a . in the URL, not sure what problems this might bring, but from UX standpoint you work with kubernetes, how would you access services across namespaces? With dots so it would be pretty obvious for a Kubernetes guy to understand this. Also I am more keen on this: An alternative is that "list all" becomes the default mode of the /system/functions/ endpoint, which can then be further refined with ?namespace=X In terms of implementation for the UI/CLI I can see how I could implement this. My input might be late here as I see that the Issue is having points, which mention the former(additional endpoint, listing namespaces etc.).

Not sure if we reached a consensus between the two propositions already? If so, I believe before touching the UI/CLI we need to introduce the endpoints/filter the namespaces, I can try to jump in with that.

[justinfx]

This sounds like a fantastic feature. I see the latest updates to the related feature ticket 511 were in December. Any update on the status of this feature?

[alexellis]

This issue shouldn't be open any more because it has been released. Regarding the update, that's not strictly correct. Checkout the documentation if you'd like to know more https://docs.openfaas.com

[alexellis]

/lock: completed

[alexellis]

For anyone landing here in 2021, there is already a link to the docs. Here's a deep link: https://docs.openfaas.com/reference/namespaces/