Namespace Improvements (Meta Issue)

mikenomitch commented 11 months ago

Proposal

Improve Namespaces to allow for truly multi-tenant Nomad cluster.

Nomad Namespaces have been used to organizationally separate teams from one another for a long time. In Nomad 1.6 Enterprise, namespaces can be tied to Node Pools. Some additional functionality could be added on top of this to make namespaces even more powerful.

Ideas:

Add nesting to namespaces - To allow for easier management of complex organizations.
Delegate more mgmt token permissions to namespace admins - To delegate things like auth method, policy, token and role creation to Nomad users who are not full-cluster admins. Other management token-only permissions could also be split out into more specific permissions.
Set eviction rules and priority defaults/max/min values per namespace - To make it easier to use preemption in Nomad. Jobs in certain namespaces could only set priority up to a specified "max" level. Some namespaces could have non-evictable jobs. This would allow clusters to have preemption on without risking worst-case scenarios of important jobs getting evicted due to other mis-configured jobs.
Isolate scheduler threads to specific namespaces - This would ensure that a single namespace could not disrupt scheduling activity on another namespace by flooding the scheduler with work.
Attach ACL policies to all jobs in a namespace - This would allow jobs in some namespaces to automatically have permissions set.

Note - I expect that if we do pursue this, some of these features would be Enterprise-only. We don't know which yet, but I don't want it to be a surprise for anybody following.

Use-cases

This would enable:

Larger Nomad clusters with multiple teams on them
More use of eviction
Safer Nomad operation without risk of one namespace throttling others
Lower security risk by splitting out management token permissions
More self-service of cluster setup, by allowing "namespace admins" to create auth roles, tokens, and more

Feedback Wanted

If you have other improvements to namespaces to suggest, let us know in a comment!

Alternatively, if you want to chat about this feature, feel free to grab a time on my calendar.

Related Issues

apollo13 commented 11 months ago

Would it be possible to clarify which of those ideas are probably going to be enterprise only?

Thinking out loud (very loud I guess) about:

Delegate more mgmt token permissions to namespace admins

This sounds a bit like you want to implement "nomad admin partitions" (the consul equivalent https://developer.hashicorp.com/consul/docs/enterprise/admin-partitions). Though one could argue that partitions + namespaces could be the same as namespace nesting?!

mikenomitch commented 11 months ago

@apollo13, I'll update once I know on the Enterprise question. If we pursue this, some of this definitely would be Enterprise-only, but I don't know where to draw the line yet. (I'll edit the ticket to clarify this as well)

You're right that there's overlap with admin partitions. I think of node pools + nested namespaces + management delecation as equivalent to a partition. We actually have the open question internally of "do we call this a partition". The upside of that is that Consul users might get the analog, the downside is that its not exactly the same and its one more concept for Nomad users to keep in their head. I lean towards just calling them nested namespaces personally. - If you or anybody else has a preference, let me know!

apollo13 commented 11 months ago

I honestly think a partition would make more sense. Eg think about authentication methods, it seems weird to add permissions for that to a namespace -- are we going to log in to a namespace then? At the end of the day it seems cleaner to me to keep the namespaces as similar as possible in oss and enterprise and then introduce a new concept (partitions) for tenancy. But that may just be me.

the-maldridge commented 11 months ago

I would actually really recommend against calling it a partition. One of my biggest annoyances when managing multiple different Hashicorp tools is that each team gave subtly different definitions to the same identifiers. If you call this a partition, can I assert that all the behaviors of consul admin partitions apply? Probably not. I'd instead try to come up with a designation that Nomad can own and rigidly define.

Also when I think in terms of a cluster scheduler and hear the word "partition" I think "failure domain" which is not really the goal here.

Changing gears and talking about things you can do with namespaces. Its my $0.02 that the ability to nest them probably shouldn't be enterprise, because that will lead to a pattern of "nesting" in the OSS world of just putting delimeters in the name. Once I've got a cluster setup with all my "nesting" done, there's zero chance I'd redo it later on because the risk to my running workloads is just too great.

The min/max priority is an interesting idea though, and to me fits in very well with the enterprise quota system. I could easily see a world in which I would allocate different amounts of each priority to different namespaces so that more preemptable priorities are "cheaper" and teams get to have more of that for free. Maybe all the way down to priority <10 where its just "free" and you can have as much as you want given you understand that you're likely getting kicked off a box pretty regularly.

The use case that pops into my mind when you start going "multi tenant" is bill-back. Even in a 10 person startup I was involved in we "billed" each application back so we could keep track of our AWS costs. The ability to nest the namespaces would have given me a cleaner solution instead of tags on jobs to work out the grouping of each application vertical, assuming this data is exposed in the ever larger firehose of the event stream.

axsuul commented 11 months ago

Can the "Jump to" in the admin UX also support namespaces too? I make heavy use of namespaces with lots of jobs that look like this

foo > app
bar > app
baz > app

So being able to search for all jobs in foo namespace and having it return

foo > app

would be extremely beneficial. Right now searching foo doesn't return anything.

hashicorp / nomad