Define Architecture and Basic Concepts for Project Mechanism

[leonpawelzik]

Define Architecture and Basic Concepts for Project Mechanism

We need to take some time to define the architecture and basic concepts for a "project" mechanism within jobq. This project mechanism should allow jobs/workloads to be part of a project, team, or something similar.

Goals:

• Define how a user can be part of a project/team.
• Ensure users can only submit to projects/teams they are part of (later stage).
• Research existing mechanisms within jobq to understand current implementation and how a project mechanism would fit.
• Draft a basic architecture diagram for the project mechanism to visualize components and interactions.

High-level design

Projects serve as a grouping mechanism (think: namespace) for workloads in the system.

Goals

As such, they serve the following goals:

• help users keep track of belonging workloads
• reduce mental workload by hiding workloads that do not matter to their current work
• define common defaults for workloads (such as Kueue queue goals)
• simplify cluster management (e.g., creating a project could automate the setup of Kubernetes resources, such as namespaces, RBAC roles, Kueue resources, security policies, ...)

Non-Goals

Explicit non-goals include:

• projects are not a security mechanism per se (e.g., a user could submit their workload into any project they are aware of)

Server-side changes

• Introduce API CRUD endpoints to manage projects (e.g., POST /projects to create a new project)
• Modify (or move under the Projects REST entity) the job submission endpoint to include the project association
• Determine Kubernetes resource associations (Kueue local queue, namespace) from project membership

Client-side changes

Modify SchedulingOptions data model to remove manual Kueue association (i.e., the queue_name attribute).

Entity-Relationship model

The following diagram illustrates the relationship between projects and other entities.

erDiagram
    Project |o..o{ User: ""
    Project {
        string name UK
        string description
        string[] tags
        bool active
    }

    Project }o--|| ClusterQueue: ""
    Project ||..|| Namespace: ""
    Namespace ||--o| LocalQueue: ""
    Namespace ||--o{ Workload: ""

    ClusterQueue ||--o{ LocalQueue: ""

Open Questions

• Where will persistent data be stored? Conceptually, the introduction of projects marks a point where the jobq backend manages fully-owned persistent state (as opposed to state managed by third parties, such as Kubernetes or Kueue). While it would be possible to attempt and store this state somewhere else (in particular, on the Kubernetes namespace object, which has a 1-1 relationship to projects in the above design), it seems inevitable to introduce a persistent data store. If we decide to do it now we'll have to go through the additional architectural considerations that come with that.
  • Since we expect neither a large data or transaction volume, an embedded database like SQLite seems like an appropriate starting point without too large impact on deployment complexity.
  • See also #43 for the same deliberation
• Will the first iteration of the project feature even require the notion of users? While it's good to keep in mind that these two concepts will eventually be linked, projects by themselves (as a grouping/namespace feature) can be useful entirely without the association to users.

[nicholasjng]

Nice!

Two high-level thoughts that came to my mind:

• I like the reference to k8s namespaces, and could totally see projects behave the same way (i.e. , a default project, listing projects, visibility)
• I also like the abstraction of the queue name, although we have to do this watertight - there should not be a project without a cluster queue (or it should be marked as unable to submit workloads), and the link between projects and queue(s) needs to be tracked somewhere. Having the user debug project<->queue association bugs is probably the worst case DX-wise.

[AdrianoKF]

• I also like the abstraction of the queue name, although we have to do this watertight - there should not be a project without a cluster queue (or it should be marked as unable to submit workloads), and the link between projects and queue(s) needs to be tracked somewhere. Having the user debug project<->queue association bugs is probably the worst case DX-wise.

That is very true. Besides having the backend check the queue association whenever submitting a workload, I could see this as a good use case for Kubernetes finalizers, which could prevent queues from being deleted while they are referenced by a project.