infra/upload-config/kubeadm phase fails saying creating config-maps is forbidden with user "Kubernetes-admin"

[teja-rafay]

What happened?

while I ran kubeadm init with addition to someother code of mine to bring up a Kubernetes cluster it fails with the below error [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace error execution phase infra/upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: configmaps is forbidden: User "kubernetes-admin" cannot create resource "configmaps" in API group "" in the namespace "kube-system" To see the stack trace of this error execute with --v=5 or higher

What did you expect to happen?

The config map should be created without any issues. and the init run should be done successfully.

How can we reproduce it (as minimally and precisely as possible)?

I wrote a go code calling the kubeadm init functions in it. when I start to initialise my cluster components it fails at the upload config phase

Anything else we need to know?

No response

Kubernetes version

$ kubectl version Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.2

Cloud provider

it's a bare metal node.

OS version

On Linux:

$ cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" $ uname -a Linux ip-172-31-6-57.us-west-2.compute.internal 3.10.0-1160.76.1.el7.x86_64 kubernetes/kubernetes#1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

containers version 1.7.18

Related plugins (CNI, CSI, ...) and versions (if applicable)

cni-plugins-linux-amd64-v1.5.1

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[HirazawaUi]

/sig cluster-lifecycle /area kubeadm

[neolit123]

/transfer kubeadm

[neolit123]

while I ran kubeadm init with addition to someother code of mine to bring up a Kubernetes cluster it fails with the below error [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace error execution phase infra/upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: configmaps is forbidden: User "kubernetes-admin" cannot create resource "configmaps" in API group "" in the namespace "kube-system" To see the stack trace of this error execute with --v=5 or higher

there was a migration in kubeadm 1.29 to have separate admin.conf and super-admin.conf https://github.com/kubernetes/kubeadm/issues/2414

you can read the design here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/4214-separate-super-user-kubeconfig

probably your admin.conf which should be part of the kubeadm:cluster-admins group is missing a clusterrolebinding to the cluster-admin role. try

kubectl create clusterrolebinding kubeadm:cluster-admins --clusterrole=cluster-admin --group=kubeadm:cluster-admins

your cluster was not migrated properly and maybe you did not call kubeadm upgrade; you should share more details on how you call commands exactly and what is the setup.

[teja-rafay]

Just to add. Mine is not an upgrade cluster. It's a standalone one at v1.30.2. My code is pretty much the same on the kubernetes level as kubeadm. Some extra things we have are come of our storage applications and other things. Let me know if you need anything else

[neolit123]

apply the fix i mentioned in the previous comment.

[neolit123]

are you calling other kubeadm phases or only upload-config?

[teja-rafay]

Almost ever single one of them. Any particular one you want me to check in particular?

[neolit123]

• i need to see the exact order of phases
• show output of the upload-config phase with the flag --v=5 added

[neolit123]

here is the default order of phases:

preflight                     Run pre-flight checks
certs                         Certificate generation
  /ca                           Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components
  /apiserver                    Generate the certificate for serving the Kubernetes API
  /apiserver-kubelet-client     Generate the certificate for the API server to connect to kubelet
  /front-proxy-ca               Generate the self-signed CA to provision identities for front proxy
  /front-proxy-client           Generate the certificate for the front proxy client
  /etcd-ca                      Generate the self-signed CA to provision identities for etcd
  /etcd-server                  Generate the certificate for serving etcd
  /etcd-peer                    Generate the certificate for etcd nodes to communicate with each other
  /etcd-healthcheck-client      Generate the certificate for liveness probes to healthcheck etcd
  /apiserver-etcd-client        Generate the certificate the apiserver uses to access etcd
  /sa                           Generate a private key for signing service account tokens along with its public key
kubeconfig                    Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
  /admin                        Generate a kubeconfig file for the admin to use and for kubeadm itself
  /super-admin                  Generate a kubeconfig file for the super-admin
  /kubelet                      Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes
  /controller-manager           Generate a kubeconfig file for the controller manager to use
  /scheduler                    Generate a kubeconfig file for the scheduler to use
etcd                          Generate static Pod manifest file for local etcd
  /local                        Generate the static Pod manifest file for a local, single-node local etcd instance
control-plane                 Generate all static Pod manifest files necessary to establish the control plane
  /apiserver                    Generates the kube-apiserver static Pod manifest
  /controller-manager           Generates the kube-controller-manager static Pod manifest
  /scheduler                    Generates the kube-scheduler static Pod manifest
kubelet-start                 Write kubelet settings and (re)start the kubelet
upload-config                 Upload the kubeadm and kubelet configuration to a ConfigMap
  /kubeadm                      Upload the kubeadm ClusterConfiguration to a ConfigMap
  /kubelet                      Upload the kubelet component config to a ConfigMap
upload-certs                  Upload certificates to kubeadm-certs
mark-control-plane            Mark a node as a control-plane
bootstrap-token               Generates bootstrap tokens used to join a node to a cluster
kubelet-finalize              Updates settings relevant to the kubelet after TLS bootstrap
  /enable-client-cert-rotation  Enable kubelet client certificate rotation
  /experimental-cert-rotation   Enable kubelet client certificate rotation (DEPRECATED: use 'enable-client-cert-rotation' instead)
addon                         Install required addons for passing conformance tests
  /coredns                      Install the CoreDNS addon to a Kubernetes cluster
  /kube-proxy                   Install the kube-proxy addon to a Kubernetes cluster
show-join-command             Show the join command for control-plane and worker node

for upload-config to work, there are a number of things that must be in place:

• a CA cert/key on disk
• a working kube-apiserver
• a valid super-admin.conf / admin.conf on disk.

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace error execution phase infra/upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: configmaps is forbidden: User "kubernetes-admin" cannot create resource "configmaps" in API group "" in the namespace "kube-system" To see the stack trace of this error execute with --v=5 or higher

the CRB should have been created in an earlier phase already.

[teja-rafay]

these are the order of phases that involve kubeadm

                        kubeadmPhases.NewCertsPhase(),
                        kubeadmPhases.NewKubeConfigPhase(),
            kubeadmPhases.NewKubeletStartPhase(),
            kubeadmPhases.NewControlPlanePhase(),
            kubeadmPhases.NewEtcdPhase(),
            kubeadmPhases.NewWaitControlPlanePhase(),
            kubeadmPhases.NewUploadConfigPhase(),
            kubeadmPhases.NewUploadCertsPhase(),
            kubeadmPhases.NewMarkControlPlanePhase(),
            kubeadmPhases.NewBootstrapTokenPhase(),
            kubeadmPhases.NewKubeletFinalizePhase(),
            kubeadmPhases.NewAddonPhase(),

[neolit123]

are you calling kubeadm source code in some sort of a forked way in your project? if so you are missing: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/init.go#L516-L545

d.client, err = kubeconfigphase.EnsureAdminClusterRoleBinding(kubeadmconstants.KubernetesDir, nil)
if err != nil {
    return nil, errors.Wrapf(err, "could not bootstrap the admin user in file %s", kubeadmconstants.AdminKubeConfigFileName)
}

[teja-rafay]

Yes @neolit123 let me try adding the missing part to my code. I'll get back to you once I test it. Thanks

[neolit123]

@teja-rafay

it should work. note, we don't provide support for users forking or embedding parts of kubeadm source code. we only provide support for users running the binary.

[teja-rafay]

Thanks @neolit123 it's working🙌🏻 Understood. will keep that in mind