Module configuration recommendation

pbochynski commented 11 months ago

Created on {2023-12-14} by Piotr Bochynski (@pbochynski).

Decision log

Name	Description
Title	Consistent default configuration for modules
Due date	2023-12-31
Status	Proposed on 2023-12-14, Accepted on 2024-01-02
Decision type	Choice
Affected decisions

Context

Each module comes with the default configuration. It is a custom resource introduced by module operator (manager). From UX perspective we need a consistent approach to that configuration. Once user get familiar with the configuration of one module it should be obvious how to configure another module.

Decision

Following rules are proposed:

location: default configuration should be placed in the module namespace (where manager is installed)
name: use default as a name
scope: use namespaced resource (not cluster wide)
single instance: if you support only one configuration use the static name and namespace, e.g. Serverless CR named default in kyma-system namespace

Consequences

Due to the different opinions the recommendations will not be forced. It means if you have a reason to use cluster wide resources - you can do it. But if you do not have any strong arguments against the recommendations please follow them.

barchw commented 11 months ago

We have collected pros and cons in @kyma-project/goat as part of the evaluation of the proposal.

In the following overview, we have labelled some items with [UX] as they evaluate how the user sees and interacts with the resources. Some of these items are opinions rather than facts as we have no data on them and they are mainly based on our assumptions of how a user should interact with our components and APIs. Also, some of these points are based on our experience with the current implementation of the user interface. Furthermore, we believe that it is important not only to look at this from a technical perspective, but also to focus on the user experience.

Cluster scoped Custom Resource		Namespaced scoped Custom Resource
Pros	Cons	Pros	Cons
Ability to use cross namespace owner references: - Allows getting all managed resources by owner reference - Allows easy to do force deletion of the module, even without a controller	[UX] Different approach than other Kyma modules	[UX] Reconciled resource is in the same namespace as manager	[UX] Module is less visible in Dashboard, as user needs to access "kyma-system" namespace, before seeing the resource
[UX] Resource is more visible to customer and the module is visible as installed, before accessing "kyma-system".	[UX] Module CR ties to Kyma is not visible by namespace, but by CR API Group only.	[UX] Resource is a clear part of Kyma, since it's in the "kyma-system" namespace.	Creation or modification of resources in "kyma-system" by customer is required, even in LM scenario
[UX] User does not need to create or edit resource in "kyma-system", which should only be managed by Kyma team	Requires additional configuration to have it displayed in Busola under "kyma-system", if we consider that to be the desired behaviour		Reconciliation for resources in different Namespaces is anyway needed, as they need to be put in "Warning" state
[UX] Configuring RBAC for creation/visibility of the module is easier (in a cluster that has users with different level of access, e.g. administrator, developer, user) This allows a more fine grained approach to configuring ClusterRoles without the need to expose access to "kyma-system"			Owner references don't bring any value, as they would only allow access to "kyma-system" resources
Monitoring module resources that are present on the cluster is easier, and doesn't need to have access to "kyma-system"			Control over Cluster scoped resources by a Namespaced scope resource feels counter-intuitive
Better solution for enforcing single module installation policy on cluster, as this does not require access to all namespaces on the cluster and doesn't need to check if resource is in "kyma-system" This allows for a more restricted RBAC configuration for the module operator as required access rights to namespaces is reduced			[UX] Customer needs to care about the namespace for module CR, when he doesn't want to use the default.
If the module installation creates other namespaces (for example Istio-system in Istio module) it feels counter intuitive that the managing resource is not higher in NS hierarchy			[UX] Restriction of module CR to "kyma-system" namespace is not obvious for the user. Because it is a namespaced resource, but it is restricted to be reconciled from a specific namespace only.
			Unnecessary fine-grained granularity: The module CR does not only manage resources in the specific namespace, but on the whole cluster.

strekm commented 11 months ago

@pbochynski can you have a look at our comment? we also run "small" migration POC and in general it is possible to migrate from cluster to namespace scope (we still got issues we need to figure out with @kyma-project/jellyfish but they seem to be resolvable) but migration is downtime, which needs to be also considered. Feel free to setup a call to discuss details.

pbochynski commented 11 months ago

Due to the different opinions the recommendations will not be forced. It means if you have a reason to use cluster wide resources - you can do it. But if you do not have any strong arguments against the recommendations please follow them.

kyma-project / community