use args --config in kube-scheduler will get resource access error: Failed to watch *v1.Node

[sonyafenge]

Repre Steps:

1. Create configmap: tc2-custom-scheduler-config in mgmt cluster using the yaml below

   cat kamaji-tc2-scheduler-configmap.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
   name: tc2-custom-scheduler-config
   namespace: kamaji-tc2
   data:
   scheduler-config.yaml: |
   apiVersion: kubescheduler.config.k8s.io/v1
   kind: KubeSchedulerConfiguration
   percentageOfNodesToScore: 5
   clientConnection:
    qps: 400
    burst: 600

2. adding these to my kamajicontrolplane/tenantcontrolplane:

   scheduler:
   extraArgs:
   - --config=/etc/kubernetes/kube-scheduler/configurations/scheduler-config.yaml
   extraVolumeMounts:
   - mountPath: /etc/kubernetes/kube-scheduler/configurations
   name: scheduler-config
   
   deployment:
   extraVolumes:
   - configMap:
      name: tc2-custom-scheduler-config
    name: scheduler-config

3. create tenantcontrolplane and kube-scheduler has this error:

   W1031 23:12:18.197842 1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kamaji-tc1:default" cannot list resource "nodes" in API group "" at the cluster scope 
   E1031 23:12:18.197867 1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kamaji-tc1:default" cannot list resource "nodes" in API group "" at the cluster scope

4. This is using user: default. if I don't use --config, I can see the user is system:kube-scheduler, which should be correct user for kube-scheduler container.

[prometherion]

When specifying a custom Scheduler plugin, the kubeconfig is mandatory, otherwise the in-cluster configuration will be used which is the ServiceAccount provided by the management cluster:

clientConnection:
  kubeconfig: "REPLACE_ME_WITH_KUBE_CONFIG_PATH"

If you specify the expected path by Kamaji, it's working as expected:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tc2-custom-scheduler-config
  namespace: kamaji-tc2
data:
  scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    percentageOfNodesToScore: 5
    clientConnection:
      kubeconfig: "/etc/kubernetes/scheduler.conf"
      qps: 400
      burst: 600

Unfortunately, we can't force Kamaji to use only the tenant kubeconfig since there could be case where the communication must be allowed between the management and the tenant cluster.

[sonyafenge]

When specifying a custom Scheduler plugin, the kubeconfig is mandatory, otherwise the in-cluster configuration will be used which is the ServiceAccount provided by the management cluster:

clientConnection:
  kubeconfig: "REPLACE_ME_WITH_KUBE_CONFIG_PATH"

If you specify the expected path by Kamaji, it's working as expected:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tc2-custom-scheduler-config
  namespace: kamaji-tc2
data:
  scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    percentageOfNodesToScore: 5
    clientConnection:
      kubeconfig: "/etc/kubernetes/scheduler.conf"
      qps: 400
      burst: 600

Unfortunately, we can't force Kamaji to use only the tenant kubeconfig since there could be case where the communication must be allowed between the management and the tenant cluster.

thanks for quick response. this do helpful and can fix the issue.