kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory

lcarva commented 2 years ago

NOTE: we suspect this is because initialization is half way done sometimes, and on next boot the generation of files is not completed

What happened:

Unable to start microshift via systemd. It panics with the following error:

I0311 17:14:22.716602       1 manager.go:112] Starting kube-apiserver
E0311 17:14:22.716772       1 kube-apiserver.go:225] kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x30c257a]

goroutine 306 [running]:
k8s.io/client-go/kubernetes.NewForConfig(0x0, 0xc00100efa8, 0x0, 0x0)
        /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/client-go/kubernetes/clientset.go:395 +0x3a
github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run.func1(0xc000728c60, 0xc0017bc180, 0xc001614000)
        /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:229 +0xa5
created by github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run
        /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:222 +0xc8

What you expected to happen:

Expected microshift to start successfully.

How to reproduce it (as minimally and precisely as possible):

Follow the steps in the Getting Started docs. Failure occurs on this step: sudo systemctl enable microshift --now

Anything else we need to know?:

A few of these warnings early in start process:

WARNING: The same type, major and minor should not be used for multiple devices.

It's unclear if these are related to the issue.

Environment:

Microshift version (use microshift version): quay.io/microshift/microshift:latest with digest sha256:efea2a587594be185554ea399f532e4d0b6b483ea9b4842c161098b554f9e54f
Hardware configuration:
OS (e.g: cat /etc/os-release): Fedora 35
Kernel (e.g. uname -a): Linux localhost.localdomain 5.16.12-200.fc35.x86_64 #1 SMP PREEMPT Wed Mar 2 19:06:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Others:

Relevant Logs

Mar 11 12:14:20 localhost.localdomain systemd[1]: Starting MicroShift Containerized...
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: 
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.750306211 -0500 EST m=+0.073820434 container create ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, io.containers.autoupdate=registry, io.buildah.version=1.23.1, io.openshift.expose-services=, build-date=2021-10-26T13:03:06.167238, io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, PODMAN_SYSTEMD_UNIT=microshift.service, architecture=x86_64, com.redhat.component=ubi8-minimal-container, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=minimal rhel8, version=8.4, vcs-type=git, name=ubi8-minimal, release=212, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Red Hat, Inc., summary=Provides the latest release of the minimal Red Hat Universal Base Image 8., vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public)
Mar 11 12:14:20 localhost.localdomain oci-umount[26772]: umounthook <debug>: prestart container_id:ea62f7801a7c rootfs:/var/lib/containers/storage/overlay/1263437b45ea09594199bc5b502a2cc2611e06699150471d4ab181db0f6c507b/merged
Mar 11 12:14:20 localhost.localdomain oci-umount[26772]: umounthook <warning>: ea62f7801a7c: path /var/lib/containers/storage from /usr/share/oci-umount/oci-umount.d/crio-umount.conf already exists
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.799207333 -0500 EST m=+0.122721541 container init ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-minimal-container, name=ubi8-minimal, vendor=Red Hat, Inc., release=212, io.containers.autoupdate=registry, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, distribution-scope=public, io.openshift.tags=minimal rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8., maintainer=Red Hat, Inc., io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, PODMAN_SYSTEMD_UNIT=microshift.service, io.openshift.expose-services=, vcs-type=git, build-date=2021-10-26T13:03:06.167238, io.buildah.version=1.23.1, version=8.4)
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.720222708 -0500 EST m=+0.043736935 image pull  quay.io/microshift/microshift:latest
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.808228662 -0500 EST m=+0.131742908 container start ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-type=git, release=212, io.buildah.version=1.23.1, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, PODMAN_SYSTEMD_UNIT=microshift.service, distribution-scope=public, build-date=2021-10-26T13:03:06.167238, maintainer=Red Hat, Inc., io.openshift.tags=minimal rhel8, io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, io.containers.autoupdate=registry, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, com.redhat.component=ubi8-minimal-container, name=ubi8-minimal, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, vendor=Red Hat, Inc., architecture=x86_64, version=8.4, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8.)
Mar 11 12:14:20 localhost.localdomain podman[26716]: ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.909574       1 openshift-apiserver.go:88] starting openshift-apiserver 192.168.86.38, args: [--config=/var/lib/microshift/resources/openshift-apiserver/config/config.yaml]
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912846       1 run.go:117] Starting Microshift
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912903       1 run.go:122] Started service-manager
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912963       1 manager.go:112] Starting etcd
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:125","msg":"configuring peer listeners","listen-peer-urls":["https://192.168.86.38:2380"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:477","msg":"starting with peer TLS","tls-info":"cert = /var/lib/microshift/certs/etcd/etcd-peer.crt, key = /var/lib/microshift/certs/etcd/etcd-peer.key, trusted-ca = /var/lib/microshift/certs/ca-bundle/ca-bundle.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:311","msg":"starting an etcd server","etcd-version":"3.4.14","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.16.12","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"localhost.localdomain","data-dir":"/var/lib/microshift/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/microshift/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.86.38:2380"],"listen-peer-urls":["https://192.168.86.38:2380"],"advertise-client-urls":["https://192.168.86.38:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":false,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":""}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"etcdserver/backend.go:80","msg":"opened backend db","path":"/var/lib/microshift/etcd/member/snap/db","took":"94.193µs"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"etcdserver/raft.go:536","msg":"restarting local member","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","commit-index":162}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:1530","msg":"129e664cf22c2231 switched to configuration voters=()"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:700","msg":"129e664cf22c2231 became follower at term 81"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:383","msg":"newRaft 129e664cf22c2231 [peers: [], term: 81, commit: 162, applied: 0, lastindex: 162, lastterm: 81]"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"warn","ts":"2022-03-11T17:14:20.916Z","caller":"auth/store.go:1366","msg":"simple token is not cryptographically signed"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.918Z","caller":"etcdserver/quota.go:98","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.919Z","caller":"etcdserver/server.go:803","msg":"starting etcd server","local-member-id":"129e664cf22c2231","local-server-version":"3.4.14","cluster-version":"to_be_decided"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"etcdserver/server.go:691","msg":"starting initial election tick advance","election-ticks":10}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"raft/raft.go:1530","msg":"129e664cf22c2231 switched to configuration voters=(1341622219669512753)"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","added-peer-id":"129e664cf22c2231","added-peer-peer-urls":["https://192.168.86.38:2380"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","cluster-version":"3.4"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /var/lib/microshift/certs/etcd/etcd-serving.crt, key = /var/lib/microshift/certs/etcd/etcd-serving.key, trusted-ca = /var/lib/microshift/certs/ca-bundle/ca-bundle.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:592","msg":"serving peer traffic","address":"192.168.86.38:2380"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:253","msg":"now serving peer/client/metrics","local-member-id":"129e664cf22c2231","initial-advertise-peer-urls":["https://192.168.86.38:2380"],"listen-peer-urls":["https://192.168.86.38:2380"],"advertise-client-urls":["https://192.168.86.38:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:799","msg":"serving metrics","address":"https://127.0.0.1:2381"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:923","msg":"129e664cf22c2231 is starting a new election at term 81"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:713","msg":"129e664cf22c2231 became candidate at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:824","msg":"129e664cf22c2231 received MsgVoteResp from 129e664cf22c2231 at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:765","msg":"129e664cf22c2231 became leader at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/node.go:325","msg":"raft.node: 129e664cf22c2231 elected leader 129e664cf22c2231 at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.716Z","caller":"etcdserver/server.go:2037","msg":"published local member to cluster through raft","local-member-id":"129e664cf22c2231","local-member-attributes":"{Name:localhost.localdomain ClientURLs:[https://192.168.86.38:2379]}","request-path":"/0/members/129e664cf22c2231/attributes","cluster-id":"1fbf54d7fd7fe23a","publish-timeout":"7s"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: I0311 17:14:22.716380       1 etcd.go:102] etcd is ready
Mar 11 12:14:22 localhost.localdomain microshift[26768]: I0311 17:14:22.716602       1 manager.go:112] Starting kube-apiserver
Mar 11 12:14:22 localhost.localdomain microshift[26768]: E0311 17:14:22.716772       1 kube-apiserver.go:225] kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory
Mar 11 12:14:22 localhost.localdomain microshift[26768]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 11 12:14:22 localhost.localdomain microshift[26768]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x30c257a]
Mar 11 12:14:22 localhost.localdomain microshift[26768]: 
Mar 11 12:14:22 localhost.localdomain microshift[26768]: goroutine 306 [running]:
Mar 11 12:14:22 localhost.localdomain microshift[26768]: k8s.io/client-go/kubernetes.NewForConfig(0x0, 0xc00100efa8, 0x0, 0x0)
Mar 11 12:14:22 localhost.localdomain microshift[26768]:         /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/client-go/kubernetes/clientset.go:395 +0x3a
Mar 11 12:14:22 localhost.localdomain microshift[26768]: github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run.func1(0xc000728c60, 0xc0017bc180, 0xc001614000)
Mar 11 12:14:22 localhost.localdomain microshift[26768]:         /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:229 +0xa5
Mar 11 12:14:22 localhost.localdomain microshift[26768]: created by github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run
Mar 11 12:14:22 localhost.localdomain microshift[26768]:         /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:222 +0xc8
Mar 11 12:14:22 localhost.localdomain podman[26789]: 2022-03-11 12:14:22.838498262 -0500 EST m=+0.069568919 container died ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift)
Mar 11 12:14:22 localhost.localdomain podman[26789]: 2022-03-11 12:14:22.868233541 -0500 EST m=+0.099304178 container remove ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, architecture=x86_64, io.openshift.expose-services=, version=8.4, io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, maintainer=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, distribution-scope=public, name=ubi8-minimal, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, vcs-type=git, io.containers.autoupdate=registry, com.redhat.component=ubi8-minimal-container, build-date=2021-10-26T13:03:06.167238, PODMAN_SYSTEMD_UNIT=microshift.service, io.openshift.tags=minimal rhel8, io.buildah.version=1.23.1, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, release=212, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8.)
Mar 11 12:14:22 localhost.localdomain systemd[1]: microshift.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 11 12:14:23 localhost.localdomain systemd[1]: microshift.service: Failed with result 'exit-code'.
Mar 11 12:14:23 localhost.localdomain systemd[1]: Failed to start MicroShift Containerized.

oglok commented 2 years ago

Hello @lcarva

What method are you using to install MicroShift? Is it RPM or containerized version?

I'd like to replicate your environment locally.

Thanks

lcarva commented 2 years ago

@oglok, the containerized version with podman. Thanks!

openshift-bot commented 2 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

ggiguash commented 2 years ago

Fixed by #794

ggiguash commented 2 years ago

/close

openshift-ci[bot] commented 2 years ago

@ggiguash: Closing this issue.

In response to [this](https://github.com/openshift/microshift/issues/630#issuecomment-1215327410): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

skylabreddy commented 1 year ago

Hi Team,

I am trying to install microshift on RHEL-8.7 and I am following Red Hat documentation

https://access.redhat.com/documentation/en-us/red_hat_build_of_microshift/4.12/html/installing/microshift-install-rpm#microshift-install-system-requirements_microshift-install-rpm

Everything went well, but while I am starting the microshift service not starting. I see below issue.

[root@microshift ~]# systemctl start microshift Job for microshift.service failed because the service did not take the steps required by its unit configuration. See "systemctl status microshift.service" and "journalctl -xe" for details. [root@microshift ~]#

Is any one can help on this issue? thanks in advance.

dhellmann commented 1 year ago

@skylabreddy can you share the output of journalctl -u microshift? It might be large, so you may want to use a separate gist or pastebin.

It would also be useful to know more about the environment where you are running. What sort of host (laptop, VM, etc.)? What networking do you have enabled?

skylabreddy commented 1 year ago

@dhellmann Here is the inputs

Microshift running on RHEL-8.7 VM
If come to network, i did not enabled any decated network for microshift,

[root@microshift ~]# journalctl -u microshift -- Logs begin at Fri 2023-03-17 20:54:09 IST, end at Fri 2023-03-17 21:09:48 IST. -- Mar 17 20:54:09 microshift systemd[1]: Starting MicroShift... Mar 17 20:54:09 microshift microshift[649415]: ??? I0317 20:54:09.958519 649415 run.go:116] Starting MicroShift Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958634 649415 certchains.go:122] [admin-kubeconfig-signer] rotate at: 2032-03-19 10:01:> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958659 649415 certchains.go:122] [admin-kubeconfig-signer admin-kubeconfig-client] rota> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958677 649415 certchains.go:122] [aggregator-signer] rotate at: 2023-11-17 10:01:49 +00> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958693 649415 certchains.go:122] [aggregator-signer aggregator-client] rotate at: 2023-> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958707 649415 certchains.go:122] [etcd-signer] rotate at: 2032-03-19 10:01:52 +0000 UTC Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958721 649415 certchains.go:122] [etcd-signer apiserver-etcd-client] rotate at: 2032-03> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958734 649415 certchains.go:122] [ingress-ca] rotate at: 2032-03-19 10:01:50 +0000 UTC Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958748 649415 certchains.go:122] [ingress-ca router-default-serving] rotate at: 2023-11> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958760 649415 certchains.go:122] [kube-apiserver-external-signer] rotate at: 2032-03-19> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958775 649415 certchains.go:122] [kube-apiserver-external-signer kube-external-serving]> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958788 649415 certchains.go:122] [kube-apiserver-localhost-signer] rotate at: 2032-03-1> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958800 649415 certchains.go:122] [kube-apiserver-localhost-signer kube-apiserver-localh> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958813 649415 certchains.go:122] [kube-apiserver-service-network-signer] rotate at: 203> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958827 649415 certchains.go:122] [kube-apiserver-service-network-signer kube-apiserver-> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958840 649415 certchains.go:122] [kube-apiserver-to-kubelet-signer] rotate at: 2023-11-> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958853 649415 certchains.go:122] [kube-apiserver-to-kubelet-signer kube-apiserver-to-ku> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958867 649415 certchains.go:122] [kube-control-plane-signer] rotate at: 2023-11-17 10:0> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958883 649415 certchains.go:122] [kube-control-plane-signer cluster-policy-controller] > Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958896 649415 certchains.go:122] [kube-control-plane-signer kube-controller-manager] ro> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958909 649415 certchains.go:122] [kube-control-plane-signer kube-scheduler] rotate at: > Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958921 649415 certchains.go:122] [kube-control-plane-signer route-controller-manager] r> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958934 649415 certchains.go:122] [kubelet-signer] rotate at: 2023-11-17 10:01:48 +0000 > Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958948 649415 certchains.go:122] [kubelet-signer kube-csr-signer] rotate at: 2023-11-17> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958963 649415 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-client] rotate> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958978 649415 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-server] rotate> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958990 649415 certchains.go:122] [service-ca] rotate at: 2032-03-19 10:01:49 +0000 UTC Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.959003 649415 certchains.go:122] [service-ca route-controller-manager-serving] rotate a> Mar 17 20:54:09 microshift microshift[649415]: ??? I0317 20:54:09.959205 649415 run.go:127] Started service-manager Mar 17 20:54:09 microshift microshift[649415]: network-configuration I0317 20:54:09.959290 649415 manager.go:114] Starting network-configuration Mar 17 20:54:09 microshift microshift[649415]: etcd I0317 20:54:09.959326 649415 manager.go:114] Starting etcd Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.959+0530","caller":"embed/etcd.go:131","msg":"configuring peer li> Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.959+0530","caller":"embed/etcd.go:481","msg":"starting with peer > Mar 17 20:54:09 microshift microshift[649415]: network-configuration I0317 20:54:09.959751 649415 netconfig.go:74] "network-configuration" is ready Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959787 649415 manager.go:114] Starting sysconfwatch-controller Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959811 649415 sysconfwatch_linux.go:89] starting sysconfwatch-contr> Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959827 649415 sysconfwatch_linux.go:95] sysconfwatch-controller is > Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.960+0530","caller":"embed/etcd.go:139","msg":"configuring client > Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.960+0530","caller":"embed/etcd.go:309","msg":"starting an etcd se> Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.961+0530","caller":"etcdserver/backend.go:81","msg":"opened backe> Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.982+0530","caller":"etcdserver/server.go:532","msg":"No snapshot > Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.998+0530","caller":"etcdserver/raft.go:483","msg":"restarting loc> Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m> Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m> Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m> lines 1-47

dhellmann commented 1 year ago

You should not need a dedicated network, but we know the CNI plugin does not work with wifi interfaces, yet. As long as your VM has a default route, you should be fine.

I do not see any obvious errors in the log, but it does not look complete, either.

How is the VM configured? How much RAM and CPU, for example?

skylabreddy commented 1 year ago

@dhellmann CPU=4 and RAM=4 and HDD is 200GB .

dhellmann commented 1 year ago

That should be plenty. It would be useful to capture more log output. Could you try running microshift outside of systemd to see if we get more information that way?

sudo /usr/bin/microshift run 2>&1 | tee /tmp/microshift.log

skylabreddy commented 1 year ago

@dhellmann Here is the log output,

[root@microshift ~]# sudo /usr/bin/microshift run 2>&1 | tee /tmp/microshift.log ??? I0318 01:18:58.624860 1154438 run.go:116] Starting MicroShift ??? E0318 01:18:58.625050 1154438 certchains.go:122] [admin-kubeconfig-signer] rotate at: 2032-03-19 10:01:47 +0000 UTC ??? E0318 01:18:58.625062 1154438 certchains.go:122] [admin-kubeconfig-signer admin-kubeconfig-client] rotate at: 2032-03-19 10:01:48 +0000 UTC ??? E0318 01:18:58.625070 1154438 certchains.go:122] [aggregator-signer] rotate at: 2023-11-17 10:01:49 +0000 UTC ??? E0318 01:18:58.625075 1154438 certchains.go:122] [aggregator-signer aggregator-client] rotate at: 2023-11-17 10:01:49 +0000 UTC ??? E0318 01:18:58.625080 1154438 certchains.go:122] [etcd-signer] rotate at: 2032-03-19 10:01:52 +0000 UTC ??? E0318 01:18:58.625084 1154438 certchains.go:122] [etcd-signer apiserver-etcd-client] rotate at: 2032-03-19 10:01:52 +0000 UTC ??? E0318 01:18:58.625089 1154438 certchains.go:122] [ingress-ca] rotate at: 2032-03-19 10:01:50 +0000 UTC ??? E0318 01:18:58.625094 1154438 certchains.go:122] [ingress-ca router-default-serving] rotate at: 2023-11-17 10:01:50 +0000 UTC ??? E0318 01:18:58.625098 1154438 certchains.go:122] [kube-apiserver-external-signer] rotate at: 2032-03-19 10:01:51 +0000 UTC ??? E0318 01:18:58.625103 1154438 certchains.go:122] [kube-apiserver-external-signer kube-external-serving] rotate at: 2023-11-17 10:01:51 +0000 UTC ??? E0318 01:18:58.625107 1154438 certchains.go:122] [kube-apiserver-localhost-signer] rotate at: 2032-03-19 10:01:51 +0000 UTC ??? E0318 01:18:58.625112 1154438 certchains.go:122] [kube-apiserver-localhost-signer kube-apiserver-localhost-serving] rotate at: 2023-11-17 10:01:51 +0000 UTC ??? E0318 01:18:58.625116 1154438 certchains.go:122] [kube-apiserver-service-network-signer] rotate at: 2032-03-19 10:01:52 +0000 UTC ??? E0318 01:18:58.625121 1154438 certchains.go:122] [kube-apiserver-service-network-signer kube-apiserver-service-network-serving] rotate at: 2023-11-17 10:01:52 +0000 UTC ??? E0318 01:18:58.625125 1154438 certchains.go:122] [kube-apiserver-to-kubelet-signer] rotate at: 2023-11-17 10:01:47 +0000 UTC ??? E0318 01:18:58.625131 1154438 certchains.go:122] [kube-apiserver-to-kubelet-signer kube-apiserver-to-kubelet-client] rotate at: 2023-11-17 10:01:47 +0000 UTC ??? E0318 01:18:58.625135 1154438 certchains.go:122] [kube-control-plane-signer] rotate at: 2023-11-17 10:01:46 +0000 UTC ??? E0318 01:18:58.625140 1154438 certchains.go:122] [kube-control-plane-signer cluster-policy-controller] rotate at: 2023-11-17 10:01:47 +0000 UTC ??? E0318 01:18:58.625145 1154438 certchains.go:122] [kube-control-plane-signer kube-controller-manager] rotate at: 2023-11-17 10:01:46 +0000 UTC ??? E0318 01:18:58.625149 1154438 certchains.go:122] [kube-control-plane-signer kube-scheduler] rotate at: 2023-11-17 10:01:46 +0000 UTC ??? E0318 01:18:58.625154 1154438 certchains.go:122] [kube-control-plane-signer route-controller-manager] rotate at: 2023-11-17 10:01:47 +0000 UTC ??? E0318 01:18:58.625158 1154438 certchains.go:122] [kubelet-signer] rotate at: 2023-11-17 10:01:48 +0000 UTC ??? E0318 01:18:58.625162 1154438 certchains.go:122] [kubelet-signer kube-csr-signer] rotate at: 2023-11-17 10:01:48 +0000 UTC ??? E0318 01:18:58.625168 1154438 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-client] rotate at: 2023-11-17 10:01:48 +0000 UTC ??? E0318 01:18:58.625173 1154438 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-server] rotate at: 2023-11-17 10:01:49 +0000 UTC ??? E0318 01:18:58.625178 1154438 certchains.go:122] [service-ca] rotate at: 2032-03-19 10:01:49 +0000 UTC ??? E0318 01:18:58.625183 1154438 certchains.go:122] [service-ca route-controller-manager-serving] rotate at: 2023-11-17 10:01:50 +0000 UTC ??? I0318 01:18:58.625352 1154438 run.go:127] Started service-manager network-configuration I0318 01:18:58.625490 1154438 manager.go:114] Starting network-configuration network-configuration I0318 01:18:58.625828 1154438 netconfig.go:74] "network-configuration" is ready etcd I0318 01:18:58.625866 1154438 manager.go:114] Starting etcd sysconfwatch-controller I0318 01:18:58.625886 1154438 manager.go:114] Starting sysconfwatch-controller sysconfwatch-controller I0318 01:18:58.625906 1154438 sysconfwatch_linux.go:89] starting sysconfwatch-controller with IP address "173.16.11.20" sysconfwatch-controller I0318 01:18:58.625916 1154438 sysconfwatch_linux.go:95] sysconfwatch-controller is ready {"level":"info","ts":"2023-03-18T01:18:58.626+0530","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://173.16.11.20:2380"]} {"level":"info","ts":"2023-03-18T01:18:58.626+0530","caller":"embed/etcd.go:481","msg":"starting with peer TLS","tls-info":"cert = /var/lib/microshift/certs/etcd-signer/etcd-peer/peer.crt, key = /var/lib/microshift/certs/etcd-signer/etcd-peer/peer.key, client-cert=, client-key=, trusted-ca = /var/lib/microshift/certs/etcd-signer/ca.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} {"level":"info","ts":"2023-03-18T01:18:58.627+0530","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://173.16.11.20:2379"]} {"level":"info","ts":"2023-03-18T01:18:58.627+0530","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.3","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.4","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"microshift","data-dir":"/var/lib/microshift/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/microshift/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://173.16.11.20:2380"],"listen-peer-urls":["https://173.16.11.20:2380"],"advertise-client-urls":["https://173.16.11.20:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://173.16.11.20:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s","max-learners":1} {"level":"info","ts":"2023-03-18T01:18:58.628+0530","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/microshift/etcd/member/snap/db","took":"257.01µs"} {"level":"info","ts":"2023-03-18T01:18:58.656+0530","caller":"etcdserver/server.go:532","msg":"No snapshot found. Recovering WAL from scratch!"} {"level":"info","ts":"2023-03-18T01:18:58.688+0530","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"69a5be14589482e1","local-member-id":"8c1bff2098e6483a","commit-index":21898} {"level":"info","ts":"2023-03-18T01:18:58.689+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a switched to configuration voters=()"} {"level":"info","ts":"2023-03-18T01:18:58.689+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became follower at term 10949"} {"level":"info","ts":"2023-03-18T01:18:58.689+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8c1bff2098e6483a [peers: [], term: 10949, commit: 21898, applied: 0, lastindex: 21898, lastterm: 10949]"} {"level":"warn","ts":"2023-03-18T01:18:58.690+0530","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} {"level":"info","ts":"2023-03-18T01:18:58.690+0530","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1} {"level":"info","ts":"2023-03-18T01:18:58.691+0530","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} {"level":"info","ts":"2023-03-18T01:18:58.691+0530","caller":"etcdserver/server.go:860","msg":"starting etcd server","local-member-id":"8c1bff2098e6483a","local-server-version":"3.5.3","cluster-version":"to_be_decided"} {"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"etcdserver/server.go:761","msg":"starting initial election tick advance","election-ticks":10} {"level":"info","ts":"2023-03-18T01:18:58.693+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a switched to configuration voters=(10095943505150363706)"} {"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"69a5be14589482e1","local-member-id":"8c1bff2098e6483a","added-peer-id":"8c1bff2098e6483a","added-peer-peer-urls":["https://173.16.11.20:2380"],"added-peer-is-learner":false} {"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"membership/cluster.go:587","msg":"set initial cluster version","cluster-id":"69a5be14589482e1","local-member-id":"8c1bff2098e6483a","cluster-version":"3.5"} {"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"} {"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:690","msg":"starting with client TLS","tls-info":"cert = /var/lib/microshift/certs/etcd-signer/etcd-serving/peer.crt, key = /var/lib/microshift/certs/etcd-signer/etcd-serving/peer.key, client-cert=, client-key=, trusted-ca = /var/lib/microshift/certs/etcd-signer/ca.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} {"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"8c1bff2098e6483a","initial-advertise-peer-urls":["https://173.16.11.20:2380"],"listen-peer-urls":["https://173.16.11.20:2380"],"advertise-client-urls":["https://173.16.11.20:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://173.16.11.20:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"]} {"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:583","msg":"serving peer traffic","address":"173.16.11.20:2380"} {"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:555","msg":"cmux::serve","address":"173.16.11.20:2380"} {"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:765","msg":"serving metrics","address":"https://127.0.0.1:2381"} {"level":"info","ts":"2023-03-18T01:19:00.290+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a is starting a new election at term 10949"} {"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became pre-candidate at term 10949"} {"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a received MsgPreVoteResp from 8c1bff2098e6483a at term 10949"} {"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became candidate at term 10950"} {"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a received MsgVoteResp from 8c1bff2098e6483a at term 10950"} {"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became leader at term 10950"} {"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8c1bff2098e6483a elected leader 8c1bff2098e6483a at term 10950"} {"level":"info","ts":"2023-03-18T01:19:00.293+0530","caller":"embed/serve.go:98","msg":"ready to serve client requests"} {"level":"info","ts":"2023-03-18T01:19:00.293+0530","caller":"etcdserver/server.go:2051","msg":"published local member to cluster through raft","local-member-id":"8c1bff2098e6483a","local-member-attributes":"{Name:microshift ClientURLs:[https://173.16.11.20:2379]}","request-path":"/0/members/8c1bff2098e6483a/attributes","cluster-id":"69a5be14589482e1","publish-timeout":"7s"} etcd I0318 01:19:00.293736 1154438 etcd.go:103] etcd is ready {"level":"info","ts":"2023-03-18T01:19:00.293+0530","caller":"embed/serve.go:98","msg":"ready to serve client requests"} {"level":"info","ts":"2023-03-18T01:19:00.296+0530","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"127.0.0.1:2379"} {"level":"info","ts":"2023-03-18T01:19:00.296+0530","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"173.16.11.20:2379"} kube-apiserver I0318 01:19:00.296302 1154438 manager.go:114] Starting kube-apiserver Flag --openshift-config has been deprecated, to be removed kube-apiserver I0318 01:19:00.298286 1154438 flags.go:64] FLAG: --add_dir_header="false" kube-apiserver I0318 01:19:00.298303 1154438 flags.go:64] FLAG: --admission-control="[]" kube-apiserver I0318 01:19:00.298317 1154438 flags.go:64] FLAG: --admission-control-config-file="" kube-apiserver I0318 01:19:00.298326 1154438 flags.go:64] FLAG: --advertise-address="" kube-apiserver I0318 01:19:00.298335 1154438 flags.go:64] FLAG: --aggregator-reject-forwarding-redirect="true" kube-apiserver I0318 01:19:00.298343 1154438 flags.go:64] FLAG: --allow-metric-labels="[]" kube-apiserver I0318 01:19:00.298357 1154438 flags.go:64] FLAG: --allow-privileged="false" kube-apiserver I0318 01:19:00.298365 1154438 flags.go:64] FLAG: --alsologtostderr="false" kube-apiserver I0318 01:19:00.298371 1154438 flags.go:64] FLAG: --anonymous-auth="true" kube-apiserver I0318 01:19:00.298377 1154438 flags.go:64] FLAG: --api-audiences="[]" kube-apiserver I0318 01:19:00.298386 1154438 flags.go:64] FLAG: --apiserver-count="1" kube-apiserver I0318 01:19:00.298394 1154438 flags.go:64] FLAG: --audit-log-batch-buffer-size="10000" kube-apiserver I0318 01:19:00.298401 1154438 flags.go:64] FLAG: --audit-log-batch-max-size="1" kube-apiserver I0318 01:19:00.298407 1154438 flags.go:64] FLAG: --audit-log-batch-max-wait="0s" kube-apiserver I0318 01:19:00.298416 1154438 flags.go:64] FLAG: --audit-log-batch-throttle-burst="0" kube-apiserver I0318 01:19:00.298422 1154438 flags.go:64] FLAG: --audit-log-batch-throttle-enable="false" kube-apiserver I0318 01:19:00.298429 1154438 flags.go:64] FLAG: --audit-log-batch-throttle-qps="0" kube-apiserver I0318 01:19:00.298437 1154438 flags.go:64] FLAG: --audit-log-compress="false" kube-apiserver I0318 01:19:00.298444 1154438 flags.go:64] FLAG: --audit-log-format="json" kube-apiserver I0318 01:19:00.298452 1154438 flags.go:64] FLAG: --audit-log-maxage="0" kube-apiserver I0318 01:19:00.298459 1154438 flags.go:64] FLAG: --audit-log-maxbackup="0" kube-apiserver I0318 01:19:00.298465 1154438 flags.go:64] FLAG: --audit-log-maxsize="0" kube-apiserver I0318 01:19:00.298471 1154438 flags.go:64] FLAG: --audit-log-mode="blocking" kube-apiserver I0318 01:19:00.298478 1154438 flags.go:64] FLAG: --audit-log-path="" kube-apiserver I0318 01:19:00.298484 1154438 flags.go:64] FLAG: --audit-log-truncate-enabled="false" kube-apiserver I0318 01:19:00.298491 1154438 flags.go:64] FLAG: --audit-log-truncate-max-batch-size="10485760" kube-apiserver I0318 01:19:00.298500 1154438 flags.go:64] FLAG: --audit-log-truncate-max-event-size="102400" kube-apiserver I0318 01:19:00.298507 1154438 flags.go:64] FLAG: --audit-log-version="audit.k8s.io/v1" kube-apiserver I0318 01:19:00.298514 1154438 flags.go:64] FLAG: --audit-policy-file="" kube-apiserver I0318 01:19:00.298520 1154438 flags.go:64] FLAG: --audit-webhook-batch-buffer-size="10000" kube-apiserver I0318 01:19:00.298526 1154438 flags.go:64] FLAG: --audit-webhook-batch-initial-backoff="10s" kube-apiserver I0318 01:19:00.298534 1154438 flags.go:64] FLAG: --audit-webhook-batch-max-size="400" kube-apiserver I0318 01:19:00.298544 1154438 flags.go:64] FLAG: --audit-webhook-batch-max-wait="30s" kube-apiserver I0318 01:19:00.298551 1154438 flags.go:64] FLAG: --audit-webhook-batch-throttle-burst="15" kube-apiserver I0318 01:19:00.298558 1154438 flags.go:64] FLAG: --audit-webhook-batch-throttle-enable="true" kube-apiserver I0318 01:19:00.298564 1154438 flags.go:64] FLAG: --audit-webhook-batch-throttle-qps="10" kube-apiserver I0318 01:19:00.298572 1154438 flags.go:64] FLAG: --audit-webhook-config-file="" kube-apiserver I0318 01:19:00.298577 1154438 flags.go:64] FLAG: --audit-webhook-initial-backoff="10s" kube-apiserver I0318 01:19:00.298584 1154438 flags.go:64] FLAG: --audit-webhook-mode="batch" kube-apiserver I0318 01:19:00.298591 1154438 flags.go:64] FLAG: --audit-webhook-truncate-enabled="false" kube-apiserver I0318 01:19:00.298597 1154438 flags.go:64] FLAG: --audit-webhook-truncate-max-batch-size="10485760" kube-apiserver I0318 01:19:00.298604 1154438 flags.go:64] FLAG: --audit-webhook-truncate-max-event-size="102400" kube-apiserver I0318 01:19:00.298611 1154438 flags.go:64] FLAG: --audit-webhook-version="audit.k8s.io/v1" kube-apiserver I0318 01:19:00.298617 1154438 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" kube-apiserver I0318 01:19:00.298632 1154438 flags.go:64] FLAG: --authentication-token-webhook-config-file="" kube-apiserver I0318 01:19:00.298646 1154438 flags.go:64] FLAG: --authentication-token-webhook-version="v1beta1" kube-apiserver I0318 01:19:00.298655 1154438 flags.go:64] FLAG: --authorization-mode="[AlwaysAllow]" kube-apiserver I0318 01:19:00.298667 1154438 flags.go:64] FLAG: --authorization-policy-file="" kube-apiserver I0318 01:19:00.298673 1154438 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" kube-apiserver I0318 01:19:00.298695 1154438 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" kube-apiserver I0318 01:19:00.298701 1154438 flags.go:64] FLAG: --authorization-webhook-config-file="" kube-apiserver I0318 01:19:00.298707 1154438 flags.go:64] FLAG: --authorization-webhook-version="v1beta1" kube-apiserver I0318 01:19:00.298715 1154438 flags.go:64] FLAG: --bind-address="0.0.0.0" kube-apiserver I0318 01:19:00.298723 1154438 flags.go:64] FLAG: --cert-dir="/var/run/kubernetes" kube-apiserver I0318 01:19:00.298730 1154438 flags.go:64] FLAG: --client-ca-file="" kube-apiserver I0318 01:19:00.298736 1154438 flags.go:64] FLAG: --cloud-config="" kube-apiserver I0318 01:19:00.298741 1154438 flags.go:64] FLAG: --cloud-provider="" kube-apiserver I0318 01:19:00.298747 1154438 flags.go:64] FLAG: --contention-profiling="false" kube-apiserver I0318 01:19:00.298753 1154438 flags.go:64] FLAG: --cors-allowed-origins="[]" kube-apiserver I0318 01:19:00.298762 1154438 flags.go:64] FLAG: --default-not-ready-toleration-seconds="300" kube-apiserver I0318 01:19:00.298769 1154438 flags.go:64] FLAG: --default-unreachable-toleration-seconds="300" kube-apiserver I0318 01:19:00.298775 1154438 flags.go:64] FLAG: --default-watch-cache-size="100" kube-apiserver I0318 01:19:00.298782 1154438 flags.go:64] FLAG: --delete-collection-workers="1" kube-apiserver I0318 01:19:00.298788 1154438 flags.go:64] FLAG: --disable-admission-plugins="[]" kube-apiserver I0318 01:19:00.298797 1154438 flags.go:64] FLAG: --disabled-metrics="[]" kube-apiserver I0318 01:19:00.298805 1154438 flags.go:64] FLAG: --egress-selector-config-file="" kube-apiserver I0318 01:19:00.298811 1154438 flags.go:64] FLAG: --enable-admission-plugins="[]" kube-apiserver I0318 01:19:00.298821 1154438 flags.go:64] FLAG: --enable-aggregator-routing="false" kube-apiserver I0318 01:19:00.298828 1154438 flags.go:64] FLAG: --enable-bootstrap-token-auth="false" kube-apiserver I0318 01:19:00.298834 1154438 flags.go:64] FLAG: --enable-garbage-collector="true" kube-apiserver I0318 01:19:00.298840 1154438 flags.go:64] FLAG: --enable-logs-handler="true" kube-apiserver I0318 01:19:00.298846 1154438 flags.go:64] FLAG: --enable-priority-and-fairness="true" kube-apiserver I0318 01:19:00.298854 1154438 flags.go:64] FLAG: --encryption-provider-config="" kube-apiserver I0318 01:19:00.298860 1154438 flags.go:64] FLAG: --endpoint-reconciler-type="lease" kube-apiserver I0318 01:19:00.298866 1154438 flags.go:64] FLAG: --etcd-cafile="" kube-apiserver I0318 01:19:00.298872 1154438 flags.go:64] FLAG: --etcd-certfile="" kube-apiserver I0318 01:19:00.298879 1154438 flags.go:64] FLAG: --etcd-compaction-interval="5m0s" kube-apiserver I0318 01:19:00.298885 1154438 flags.go:64] FLAG: --etcd-count-metric-poll-period="1m0s" kube-apiserver I0318 01:19:00.298892 1154438 flags.go:64] FLAG: --etcd-db-metric-poll-interval="30s" kube-apiserver I0318 01:19:00.298898 1154438 flags.go:64] FLAG: --etcd-healthcheck-timeout="2s" kube-apiserver I0318 01:19:00.298905 1154438 flags.go:64] FLAG: --etcd-keyfile="" kube-apiserver I0318 01:19:00.298911 1154438 flags.go:64] FLAG: --etcd-prefix="/registry" kube-apiserver I0318 01:19:00.298917 1154438 flags.go:64] FLAG: --etcd-readycheck-timeout="2s" kube-apiserver I0318 01:19:00.298923 1154438 flags.go:64] FLAG: --etcd-servers="[]" kube-apiserver I0318 01:19:00.298932 1154438 flags.go:64] FLAG: --etcd-servers-overrides="[]" kube-apiserver I0318 01:19:00.298979 1154438 flags.go:64] FLAG: --event-ttl="1h0m0s" kube-apiserver I0318 01:19:00.298987 1154438 flags.go:64] FLAG: --external-hostname="" kube-apiserver I0318 01:19:00.298992 1154438 flags.go:64] FLAG: --feature-gates="" kube-apiserver I0318 01:19:00.299004 1154438 flags.go:64] FLAG: --goaway-chance="0" kube-apiserver I0318 01:19:00.299014 1154438 flags.go:64] FLAG: --help="false" kube-apiserver I0318 01:19:00.299020 1154438 flags.go:64] FLAG: --http2-max-streams-per-connection="0" kube-apiserver I0318 01:19:00.299026 1154438 flags.go:64] FLAG: --identity-lease-duration-seconds="3600" kube-apiserver I0318 01:19:00.299033 1154438 flags.go:64] FLAG: --identity-lease-renew-interval-seconds="10" kube-apiserver I0318 01:19:00.299056 1154438 flags.go:64] FLAG: --kubelet-certificate-authority="" kube-apiserver I0318 01:19:00.299063 1154438 flags.go:64] FLAG: --kubelet-client-certificate="" kube-apiserver I0318 01:19:00.299068 1154438 flags.go:64] FLAG: --kubelet-client-key="" kube-apiserver I0318 01:19:00.299074 1154438 flags.go:64] FLAG: --kubelet-port="10250" kube-apiserver I0318 01:19:00.299083 1154438 flags.go:64] FLAG: --kubelet-preferred-address-types="[Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP]" kube-apiserver I0318 01:19:00.299095 1154438 flags.go:64] FLAG: --kubelet-read-only-port="10255" kube-apiserver I0318 01:19:00.299102 1154438 flags.go:64] FLAG: --kubelet-timeout="5s" kube-apiserver I0318 01:19:00.299108 1154438 flags.go:64] FLAG: --kubernetes-service-node-port="0" kube-apiserver I0318 01:19:00.299114 1154438 flags.go:64] FLAG: --lease-reuse-duration-seconds="60" kube-apiserver I0318 01:19:00.299120 1154438 flags.go:64] FLAG: --livez-grace-period="0s" kube-apiserver I0318 01:19:00.299126 1154438 flags.go:64] FLAG: --log-flush-frequency="5s" kube-apiserver I0318 01:19:00.299133 1154438 flags.go:64] FLAG: --log_backtrace_at=":0" kube-apiserver I0318 01:19:00.299142 1154438 flags.go:64] FLAG: --log_dir="" kube-apiserver I0318 01:19:00.299150 1154438 flags.go:64] FLAG: --log_file="" kube-apiserver I0318 01:19:00.299156 1154438 flags.go:64] FLAG: --log_file_max_size="1800" kube-apiserver I0318 01:19:00.299163 1154438 flags.go:64] FLAG: --logging-format="text" kube-apiserver I0318 01:19:00.299170 1154438 flags.go:64] FLAG: --logtostderr="true" kube-apiserver I0318 01:19:00.299177 1154438 flags.go:64] FLAG: --master-service-namespace="default" kube-apiserver I0318 01:19:00.299183 1154438 flags.go:64] FLAG: --max-connection-bytes-per-sec="0" kube-apiserver I0318 01:19:00.299190 1154438 flags.go:64] FLAG: --max-mutating-requests-inflight="200" kube-apiserver I0318 01:19:00.299197 1154438 flags.go:64] FLAG: --max-requests-inflight="400" kube-apiserver I0318 01:19:00.299203 1154438 flags.go:64] FLAG: --min-request-timeout="1800" kube-apiserver I0318 01:19:00.299211 1154438 flags.go:64] FLAG: --oidc-ca-file="" kube-apiserver I0318 01:19:00.299217 1154438 flags.go:64] FLAG: --oidc-client-id="" kube-apiserver I0318 01:19:00.299223 1154438 flags.go:64] FLAG: --oidc-groups-claim="" kube-apiserver I0318 01:19:00.299230 1154438 flags.go:64] FLAG: --oidc-groups-prefix="" kube-apiserver I0318 01:19:00.299235 1154438 flags.go:64] FLAG: --oidc-issuer-url="" kube-apiserver I0318 01:19:00.299241 1154438 flags.go:64] FLAG: --oidc-required-claim="" kube-apiserver I0318 01:19:00.299250 1154438 flags.go:64] FLAG: --oidc-signing-algs="[RS256]" kube-apiserver I0318 01:19:00.299261 1154438 flags.go:64] FLAG: --oidc-username-claim="sub" kube-apiserver I0318 01:19:00.299267 1154438 flags.go:64] FLAG: --oidc-username-prefix="" kube-apiserver I0318 01:19:00.299273 1154438 flags.go:64] FLAG: --one_output="false" kube-apiserver I0318 01:19:00.299280 1154438 flags.go:64] FLAG: --openshift-config="/tmp/kube-apiserver-config-3180920900.yaml" kube-apiserver I0318 01:19:00.299287 1154438 flags.go:64] FLAG: --permit-address-sharing="false" kube-apiserver I0318 01:19:00.299294 1154438 flags.go:64] FLAG: --permit-port-sharing="false" kube-apiserver I0318 01:19:00.299300 1154438 flags.go:64] FLAG: --profiling="true" kube-apiserver I0318 01:19:00.299307 1154438 flags.go:64] FLAG: --proxy-client-cert-file="" kube-apiserver I0318 01:19:00.299313 1154438 flags.go:64] FLAG: --proxy-client-key-file="" kube-apiserver I0318 01:19:00.299319 1154438 flags.go:64] FLAG: --request-timeout="1m0s" kube-apiserver I0318 01:19:00.299325 1154438 flags.go:64] FLAG: --requestheader-allowed-names="[]" kube-apiserver I0318 01:19:00.299333 1154438 flags.go:64] FLAG: --requestheader-client-ca-file="" kube-apiserver I0318 01:19:00.299339 1154438 flags.go:64] FLAG: --requestheader-extra-headers-prefix="[]" kube-apiserver I0318 01:19:00.299348 1154438 flags.go:64] FLAG: --requestheader-group-headers="[]" kube-apiserver I0318 01:19:00.299356 1154438 flags.go:64] FLAG: --requestheader-username-headers="[]" kube-apiserver I0318 01:19:00.299366 1154438 flags.go:64] FLAG: --runtime-config="" kube-apiserver I0318 01:19:00.299375 1154438 flags.go:64] FLAG: --secure-port="6443" kube-apiserver I0318 01:19:00.299381 1154438 flags.go:64] FLAG: --send-retry-after-while-not-ready-once="false" kube-apiserver I0318 01:19:00.299388 1154438 flags.go:64] FLAG: --service-account-extend-token-expiration="true" kube-apiserver I0318 01:19:00.299394 1154438 flags.go:64] FLAG: --service-account-issuer="[]" kube-apiserver I0318 01:19:00.299405 1154438 flags.go:64] FLAG: --service-account-jwks-uri="" kube-apiserver I0318 01:19:00.299411 1154438 flags.go:64] FLAG: --service-account-key-file="[]" kube-apiserver I0318 01:19:00.299421 1154438 flags.go:64] FLAG: --service-account-lookup="true" kube-apiserver I0318 01:19:00.299428 1154438 flags.go:64] FLAG: --service-account-max-token-expiration="0s" kube-apiserver I0318 01:19:00.299434 1154438 flags.go:64] FLAG: --service-account-signing-key-file="" kube-apiserver I0318 01:19:00.299440 1154438 flags.go:64] FLAG: --service-cluster-ip-range="" kube-apiserver I0318 01:19:00.299446 1154438 flags.go:64] FLAG: --service-node-port-range="30000-32767" kube-apiserver I0318 01:19:00.299457 1154438 flags.go:64] FLAG: --show-hidden-metrics-for-version="" kube-apiserver I0318 01:19:00.299463 1154438 flags.go:64] FLAG: --shutdown-delay-duration="0s" kube-apiserver I0318 01:19:00.299469 1154438 flags.go:64] FLAG: --shutdown-send-retry-after="false" kube-apiserver I0318 01:19:00.299475 1154438 flags.go:64] FLAG: --skip_headers="false" kube-apiserver I0318 01:19:00.299481 1154438 flags.go:64] FLAG: --skip_log_headers="false" kube-apiserver I0318 01:19:00.299488 1154438 flags.go:64] FLAG: --stderrthreshold="2" kube-apiserver I0318 01:19:00.299494 1154438 flags.go:64] FLAG: --storage-backend="" kube-apiserver I0318 01:19:00.299500 1154438 flags.go:64] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf" kube-apiserver I0318 01:19:00.299508 1154438 flags.go:64] FLAG: --strict-transport-security-directives="[]" kube-apiserver I0318 01:19:00.299517 1154438 flags.go:64] FLAG: --tls-cert-file="" kube-apiserver I0318 01:19:00.299523 1154438 flags.go:64] FLAG: --tls-cipher-suites="[]" kube-apiserver I0318 01:19:00.299532 1154438 flags.go:64] FLAG: --tls-min-version="" kube-apiserver I0318 01:19:00.299539 1154438 flags.go:64] FLAG: --tls-private-key-file="" kube-apiserver I0318 01:19:00.299545 1154438 flags.go:64] FLAG: --tls-sni-cert-key="[]" kube-apiserver I0318 01:19:00.299554 1154438 flags.go:64] FLAG: --token-auth-file="" kube-apiserver I0318 01:19:00.299560 1154438 flags.go:64] FLAG: --tracing-config-file="" kube-apiserver I0318 01:19:00.299566 1154438 flags.go:64] FLAG: --v="2" kube-apiserver I0318 01:19:00.299575 1154438 flags.go:64] FLAG: --version="false" kube-apiserver I0318 01:19:00.299584 1154438 flags.go:64] FLAG: --vmodule="" kube-apiserver I0318 01:19:00.299592 1154438 flags.go:64] FLAG: --watch-cache="true" kube-apiserver I0318 01:19:00.299599 1154438 flags.go:64] FLAG: --watch-cache-sizes="[]" kube-apiserver I0318 01:19:00.299649 1154438 plugins.go:84] "Registered admission plugin" plugin="authorization.openshift.io/RestrictSubjectBindings" kube-apiserver I0318 01:19:00.299669 1154438 plugins.go:84] "Registered admission plugin" plugin="route.openshift.io/RouteHostAssignment" kube-apiserver I0318 01:19:00.299682 1154438 plugins.go:84] "Registered admission plugin" plugin="image.openshift.io/ImagePolicy" kube-apiserver I0318 01:19:00.299694 1154438 plugins.go:84] "Registered admission plugin" plugin="route.openshift.io/IngressAdmission" kube-apiserver I0318 01:19:00.299713 1154438 plugins.go:84] "Registered admission plugin" plugin="autoscaling.openshift.io/ManagementCPUsOverride" kube-apiserver I0318 01:19:00.299726 1154438 plugins.go:84] "Registered admission plugin" plugin="scheduling.openshift.io/OriginPodNodeEnvironment" kube-apiserver I0318 01:19:00.299738 1154438 plugins.go:84] "Registered admission plugin" plugin="autoscaling.openshift.io/ClusterResourceOverride" kube-apiserver I0318 01:19:00.299750 1154438 plugins.go:84] "Registered admission plugin" plugin="quota.openshift.io/ClusterResourceQuota" kube-apiserver I0318 01:19:00.299776 1154438 plugins.go:84] "Registered admission plugin" plugin="autoscaling.openshift.io/RunOnceDuration" kube-apiserver I0318 01:19:00.299786 1154438 plugins.go:84] "Registered admission plugin" plugin="scheduling.openshift.io/PodNodeConstraints" kube-apiserver I0318 01:19:00.299796 1154438 plugins.go:84] "Registered admission plugin" plugin="security.openshift.io/SecurityContextConstraint" kube-apiserver I0318 01:19:00.299805 1154438 plugins.go:84] "Registered admission plugin" plugin="security.openshift.io/SCCExecRestrictions" kube-apiserver I0318 01:19:00.299814 1154438 plugins.go:84] "Registered admission plugin" plugin="network.openshift.io/ExternalIPRanger" kube-apiserver I0318 01:19:00.299824 1154438 plugins.go:84] "Registered admission plugin" plugin="network.openshift.io/RestrictedEndpointsAdmission" kube-apiserver I0318 01:19:00.299834 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateAPIServer" kube-apiserver I0318 01:19:00.299843 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateAuthentication" kube-apiserver I0318 01:19:00.299853 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateFeatureGate" kube-apiserver I0318 01:19:00.299862 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateConsole" kube-apiserver I0318 01:19:00.299871 1154438 plugins.go:84] "Registered admission plugin" plugin="operator.openshift.io/ValidateDNS" kube-apiserver I0318 01:19:00.299886 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateImage" kube-apiserver I0318 01:19:00.299896 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateOAuth" kube-apiserver I0318 01:19:00.299907 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateProject" kube-apiserver I0318 01:19:00.299918 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/DenyDeleteClusterConfiguration" kube-apiserver I0318 01:19:00.299929 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateScheduler" kube-apiserver I0318 01:19:00.299938 1154438 plugins.go:84] "Registered admission plugin" plugin="operator.openshift.io/ValidateKubeControllerManager" kube-apiserver I0318 01:19:00.299949 1154438 plugins.go:84] "Registered admission plugin" plugin="quota.openshift.io/ValidateClusterResourceQuota" kube-apiserver I0318 01:19:00.299959 1154438 plugins.go:84] "Registered admission plugin" plugin="security.openshift.io/ValidateSecurityContextConstraints" kube-apiserver I0318 01:19:00.299969 1154438 plugins.go:84] "Registered admission plugin" plugin="authorization.openshift.io/ValidateRoleBindingRestriction" kube-apiserver I0318 01:19:00.299978 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateNetwork" kube-apiserver I0318 01:19:00.299988 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateAPIRequestCount" kube-apiserver I0318 01:19:00.299997 1154438 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/RestrictExtremeWorkerLatencyProfile" kube-apiserver I0318 01:19:00.300007 1154438 plugins.go:84] "Registered admission plugin" plugin="security.openshift.io/DefaultSecurityContextConstraints" kube-apiserver I0318 01:19:00.300016 1154438 plugins.go:84] "Registered admission plugin" plugin="route.openshift.io/ValidateRoute" kube-apiserver I0318 01:19:00.300025 1154438 plugins.go:84] "Registered admission plugin" plugin="route.openshift.io/DefaultRoute" Flag --openshift-config has been deprecated, to be removed Flag --enable-logs-handler has been deprecated, This flag will be removed in v1.19 Flag --kubelet-read-only-port has been deprecated, kubelet-read-only-port is deprecated and will be removed. kube-apiserver I0318 01:19:00.303295 1154438 flags.go:64] FLAG: --add_dir_header="false" kube-apiserver I0318 01:19:00.303307 1154438 flags.go:64] FLAG: --admission-control="[]" kube-apiserver I0318 01:19:00.303316 1154438 flags.go:64] FLAG: --admission-control-config-file="/tmp/kubeapiserver-admission-config.yaml3587889150" kube-apiserver I0318 01:19:00.303324 1154438 flags.go:64] FLAG: --advertise-address="10.43.0.1" kube-apiserver I0318 01:19:00.303330 1154438 flags.go:64] FLAG: --aggregator-reject-forwarding-redirect="true" kube-apiserver I0318 01:19:00.303336 1154438 flags.go:64] FLAG: --allow-metric-labels="[]" kube-apiserver I0318 01:19:00.303345 1154438 flags.go:64] FLAG: --allow-privileged="true" kube-apiserver I0318 01:19:00.303351 1154438 flags.go:64] FLAG: --alsologtostderr="false" kube-apiserver I0318 01:19:00.303356 1154438 flags.go:64] FLAG: --anonymous-auth="true" kube-apiserver I0318 01:19:00.303361 1154438 flags.go:64] FLAG: --api-audiences="[https://kubernetes.default.svc]" kube-apiserver I0318 01:19:00.303370 1154438 flags.go:64] FLAG: --apiserver-count="1" kube-apiserver I0318 01:19:00.303376 1154438 flags.go:64] FLAG: --audit-log-batch-buffer-size="10000" kube-apiserver I0318 01:19:00.303382 1154438 flags.go:64] FLAG: --audit-log-batch-max-size="1" kube-apiserver I0318 01:19:00.303387 1154438 flags.go:64] FLAG: --audit-log-batch-max-wait="0s" kube-apiserver I0318 01:19:00.303392 1154438 flags.go:64] FLAG: --audit-log-batch-throttle-burst="0" kube-apiserver I0318 01:19:00.303397 1154438 flags.go:64] FLAG: --audit-log-batch-throttle-enable="false" kube-apiserver I0318 01:19:00.303402 1154438 flags.go:64] FLAG: --audit-log-batch-throttle-qps="0" kube-apiserver I0318 01:19:00.303408 1154438 flags.go:64] FLAG: --audit-log-compress="false" kube-apiserver I0318 01:19:00.303413 1154438 flags.go:64] FLAG: --audit-log-format="json" kube-apiserver I0318 01:19:00.303419 1154438 flags.go:64] FLAG: --audit-log-maxage="0" kube-apiserver I0318 01:19:00.303424 1154438 flags.go:64] FLAG: --audit-log-maxbackup="10" kube-apiserver I0318 01:19:00.303429 1154438 flags.go:64] FLAG: --audit-log-maxsize="100" kube-apiserver I0318 01:19:00.303437 1154438 flags.go:64] FLAG: --audit-log-mode="blocking" kube-apiserver I0318 01:19:00.303442 1154438 flags.go:64] FLAG: --audit-log-path="/var/log/kube-apiserver/audit.log" kube-apiserver I0318 01:19:00.303448 1154438 flags.go:64] FLAG: --audit-log-truncate-enabled="false" kube-apiserver I0318 01:19:00.303453 1154438 flags.go:64] FLAG: --audit-log-truncate-max-batch-size="10485760" kube-apiserver I0318 01:19:00.303459 1154438 flags.go:64] FLAG: --audit-log-truncate-max-event-size="102400" kube-apiserver I0318 01:19:00.303464 1154438 flags.go:64] FLAG: --audit-log-version="audit.k8s.io/v1" kube-apiserver I0318 01:19:00.303470 1154438 flags.go:64] FLAG: --audit-policy-file="/var/lib/microshift/resources/kube-apiserver-audit-policies/default.yaml" kube-apiserver I0318 01:19:00.303477 1154438 flags.go:64] FLAG: --audit-webhook-batch-buffer-size="10000" kube-apiserver I0318 01:19:00.303482 1154438 flags.go:64] FLAG: --audit-webhook-batch-initial-backoff="10s" kube-apiserver I0318 01:19:00.303488 1154438 flags.go:64] FLAG: --audit-webhook-batch-max-size="400" kube-apiserver I0318 01:19:00.303493 1154438 flags.go:64] FLAG: --audit-webhook-batch-max-wait="30s" kube-apiserver I0318 01:19:00.303498 1154438 flags.go:64] FLAG: --audit-webhook-batch-throttle-burst="15" kube-apiserver I0318 01:19:00.303503 1154438 flags.go:64] FLAG: --audit-webhook-batch-throttle-enable="true" kube-apiserver I0318 01:19:00.303508 1154438 flags.go:64] FLAG: --audit-webhook-batch-throttle-qps="10" kube-apiserver I0318 01:19:00.303515 1154438 flags.go:64] FLAG: --audit-webhook-config-file="" kube-apiserver I0318 01:19:00.303521 1154438 flags.go:64] FLAG: --audit-webhook-initial-backoff="10s" kube-apiserver I0318 01:19:00.303526 1154438 flags.go:64] FLAG: --audit-webhook-mode="batch" kube-apiserver I0318 01:19:00.303531 1154438 flags.go:64] FLAG: --audit-webhook-truncate-enabled="false" kube-apiserver I0318 01:19:00.303536 1154438 flags.go:64] FLAG: --audit-webhook-truncate-max-batch-size="10485760" kube-apiserver I0318 01:19:00.303542 1154438 flags.go:64] FLAG: --audit-webhook-truncate-max-event-size="102400" kube-apiserver I0318 01:19:00.303547 1154438 flags.go:64] FLAG: --audit-webhook-version="audit.k8s.io/v1" kube-apiserver I0318 01:19:00.303553 1154438 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" kube-apiserver I0318 01:19:00.303559 1154438 flags.go:64] FLAG: --authentication-token-webhook-config-file="" kube-apiserver I0318 01:19:00.303564 1154438 flags.go:64] FLAG: --authentication-token-webhook-version="v1beta1" kube-apiserver I0318 01:19:00.303568 1154438 flags.go:64] FLAG: --authorization-mode="[Scope,SystemMasters,RBAC,Node]" kube-apiserver I0318 01:19:00.303578 1154438 flags.go:64] FLAG: --authorization-policy-file="" kube-apiserver I0318 01:19:00.303583 1154438 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" kube-apiserver I0318 01:19:00.303588 1154438 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" kube-apiserver I0318 01:19:00.303594 1154438 flags.go:64] FLAG: --authorization-webhook-config-file="" kube-apiserver I0318 01:19:00.303598 1154438 flags.go:64] FLAG: --authorization-webhook-version="v1beta1" kube-apiserver I0318 01:19:00.303603 1154438 flags.go:64] FLAG: --bind-address="0.0.0.0" kube-apiserver I0318 01:19:00.303609 1154438 flags.go:64] FLAG: --cert-dir="/var/run/kubernetes" kube-apiserver I0318 01:19:00.303614 1154438 flags.go:64] FLAG: --client-ca-file="/var/lib/microshift/certs/ca-bundle/client-ca.crt" kube-apiserver I0318 01:19:00.303621 1154438 flags.go:64] FLAG: --cloud-config="" kube-apiserver I0318 01:19:00.303626 1154438 flags.go:64] FLAG: --cloud-provider="" kube-apiserver I0318 01:19:00.303631 1154438 flags.go:64] FLAG: --contention-profiling="false" kube-apiserver I0318 01:19:00.303639 1154438 flags.go:64] FLAG: --cors-allowed-origins="[//127\.0\.0\.1(:|$),//localhost(:|$)]" kube-apiserver I0318 01:19:00.303648 1154438 flags.go:64] FLAG: --default-not-ready-toleration-seconds="300" kube-apiserver I0318 01:19:00.303659 1154438 flags.go:64] FLAG: --default-unreachable-toleration-seconds="300" kube-apiserver I0318 01:19:00.303670 1154438 flags.go:64] FLAG: --default-watch-cache-size="100" kube-apiserver I0318 01:19:00.303675 1154438 flags.go:64] FLAG: --delete-collection-workers="1" kube-apiserver I0318 01:19:00.303680 1154438 flags.go:64] FLAG: --disable-admission-plugins="[authorization.openshift.io/RestrictSubjectBindings,authorization.openshift.io/ValidateRoleBindingRestriction,autoscaling.openshift.io/ManagementCPUsOverride,config.openshift.io/DenyDeleteClusterConfiguration,config.openshift.io/ValidateAPIServer,config.openshift.io/ValidateAuthentication,config.openshift.io/ValidateConsole,config.openshift.io/ValidateFeatureGate,config.openshift.io/ValidateImage,config.openshift.io/ValidateOAuth,config.openshift.io/ValidateProject,config.openshift.io/ValidateScheduler,image.openshift.io/ImagePolicy,quota.openshift.io/ClusterResourceQuota,quota.openshift.io/ValidateClusterResourceQuota]" kube-apiserver I0318 01:19:00.303715 1154438 flags.go:64] FLAG: --disabled-metrics="[]" kube-apiserver I0318 01:19:00.303722 1154438 flags.go:64] FLAG: --egress-selector-config-file="" kube-apiserver I0318 01:19:00.303619 1154438 kube-apiserver.go:323] "kube-apiserver" not yet ready: Get "https://127.0.0.1:6443/readyz": dial tcp 127.0.0.1:6443: connect: connection refused kube-apiserver I0318 01:19:00.303728 1154438 flags.go:64] FLAG: --enable-admission-plugins="[CertificateApproval,CertificateSigning,CertificateSubjectRestriction,DefaultIngressClass,DefaultStorageClass,DefaultTolerationSeconds,LimitRanger,MutatingAdmissionWebhook,NamespaceLifecycle,NodeRestriction,OwnerReferencesPermissionEnforcement,PersistentVolumeClaimResize,PersistentVolumeLabel,PodNodeSelector,PodTolerationRestriction,Priority,ResourceQuota,RuntimeClass,ServiceAccount,StorageObjectInUseProtection,TaintNodesByCondition,ValidatingAdmissionWebhook,network.openshift.io/ExternalIPRanger,network.openshift.io/RestrictedEndpointsAdmission,route.openshift.io/IngressAdmission,scheduling.openshift.io/OriginPodNodeEnvironment,security.openshift.io/DefaultSecurityContextConstraints,security.openshift.io/SCCExecRestrictions,security.openshift.io/SecurityContextConstraint,security.openshift.io/ValidateSecurityContextConstraints]" kube-apiserver I0318 01:19:00.303763 1154438 flags.go:64] FLAG: --enable-aggregator-routing="true" kube-apiserver I0318 01:19:00.303768 1154438 flags.go:64] FLAG: --enable-bootstrap-token-auth="false" kube-apiserver I0318 01:19:00.303773 1154438 flags.go:64] FLAG: --enable-garbage-collector="true" kube-apiserver I0318 01:19:00.303778 1154438 flags.go:64] FLAG: --enable-logs-handler="false" kube-apiserver I0318 01:19:00.303783 1154438 flags.go:64] FLAG: --enable-priority-and-fairness="true" kube-apiserver I0318 01:19:00.303788 1154438 flags.go:64] FLAG: --encryption-provider-config="" kube-apiserver I0318 01:19:00.303794 1154438 flags.go:64] FLAG: --endpoint-reconciler-type="lease" kube-apiserver I0318 01:19:00.303799 1154438 flags.go:64] FLAG: --etcd-cafile="/var/lib/microshift/certs/etcd-signer/ca.crt" kube-apiserver I0318 01:19:00.303804 1154438 flags.go:64] FLAG: --etcd-certfile="/var/lib/microshift/certs/etcd-signer/apiserver-etcd-client/client.crt" kube-apiserver I0318 01:19:00.303811 1154438 flags.go:64] FLAG: --etcd-compaction-interval="5m0s" kube-apiserver I0318 01:19:00.303817 1154438 flags.go:64] FLAG: --etcd-count-metric-poll-period="1m0s" kube-apiserver I0318 01:19:00.303822 1154438 flags.go:64] FLAG: --etcd-db-metric-poll-interval="30s" kube-apiserver I0318 01:19:00.303827 1154438 flags.go:64] FLAG: --etcd-healthcheck-timeout="2s" kube-apiserver I0318 01:19:00.303832 1154438 flags.go:64] FLAG: --etcd-keyfile="/var/lib/microshift/certs/etcd-signer/apiserver-etcd-client/client.key" kube-apiserver I0318 01:19:00.303840 1154438 flags.go:64] FLAG: --etcd-prefix="kubernetes.io" kube-apiserver I0318 01:19:00.303846 1154438 flags.go:64] FLAG: --etcd-readycheck-timeout="2s" kube-apiserver I0318 01:19:00.303851 1154438 flags.go:64] FLAG: --etcd-servers="[https://127.0.0.1:2379]" kube-apiserver I0318 01:19:00.303859 1154438 flags.go:64] FLAG: --etcd-servers-overrides="[]" kube-apiserver I0318 01:19:00.303867 1154438 flags.go:64] FLAG: --event-ttl="3h0m0s" kube-apiserver I0318 01:19:00.303872 1154438 flags.go:64] FLAG: --external-hostname="" kube-apiserver I0318 01:19:00.303877 1154438 flags.go:64] FLAG: --feature-gates="" kube-apiserver I0318 01:19:00.303884 1154438 flags.go:64] FLAG: --goaway-chance="0" kube-apiserver I0318 01:19:00.303891 1154438 flags.go:64] FLAG: --help="false" kube-apiserver I0318 01:19:00.303896 1154438 flags.go:64] FLAG: --http2-max-streams-per-connection="2000" kube-apiserver I0318 01:19:00.303901 1154438 flags.go:64] FLAG: --identity-lease-duration-seconds="3600" kube-apiserver I0318 01:19:00.303906 1154438 flags.go:64] FLAG: --identity-lease-renew-interval-seconds="10" kube-apiserver I0318 01:19:00.303912 1154438 flags.go:64] FLAG: --kubelet-certificate-authority="/var/lib/microshift/certs/kubelet-csr-signer-signer/csr-signer/ca-bundle.crt" kube-apiserver I0318 01:19:00.303919 1154438 flags.go:64] FLAG: --kubelet-client-certificate="/var/lib/microshift/certs/kube-apiserver-to-kubelet-client-signer/kube-apiserver-to-kubelet-client/client.crt" kube-apiserver I0318 01:19:00.303927 1154438 flags.go:64] FLAG: --kubelet-client-key="/var/lib/microshift/certs/kube-apiserver-to-kubelet-client-signer/kube-apiserver-to-kubelet-client/client.key" kube-apiserver I0318 01:19:00.303934 1154438 flags.go:64] FLAG: --kubelet-port="10250" kube-apiserver I0318 01:19:00.303940 1154438 flags.go:64] FLAG: --kubelet-preferred-address-types="[InternalIP]" kube-apiserver I0318 01:19:00.303947 1154438 flags.go:64] FLAG: --kubelet-read-only-port="0" kube-apiserver I0318 01:19:00.303952 1154438 flags.go:64] FLAG: --kubelet-timeout="5s" kube-apiserver I0318 01:19:00.303959 1154438 flags.go:64] FLAG: --kubernetes-service-node-port="0" kube-apiserver I0318 01:19:00.303964 1154438 flags.go:64] FLAG: --lease-reuse-duration-seconds="60" kube-apiserver I0318 01:19:00.303969 1154438 flags.go:64] FLAG: --livez-grace-period="0s" kube-apiserver I0318 01:19:00.303974 1154438 flags.go:64] FLAG: --log-flush-frequency="5s" kube-apiserver I0318 01:19:00.303985 1154438 flags.go:64] FLAG: --log_backtrace_at=":0" kube-apiserver I0318 01:19:00.303994 1154438 flags.go:64] FLAG: --log_dir="" kube-apiserver I0318 01:19:00.304000 1154438 flags.go:64] FLAG: --log_file="" kube-apiserver I0318 01:19:00.304004 1154438 flags.go:64] FLAG: --log_file_max_size="1800" kube-apiserver I0318 01:19:00.304010 1154438 flags.go:64] FLAG: --logging-format="text" kube-apiserver I0318 01:19:00.304015 1154438 flags.go:64] FLAG: --logtostderr="true" kube-apiserver I0318 01:19:00.304020 1154438 flags.go:64] FLAG: --master-service-namespace="default" kube-apiserver I0318 01:19:00.304025 1154438 flags.go:64] FLAG: --max-connection-bytes-per-sec="0" kube-apiserver I0318 01:19:00.304031 1154438 flags.go:64] FLAG: --max-mutating-requests-inflight="1000" kube-apiserver I0318 01:19:00.304064 1154438 flags.go:64] FLAG: --max-requests-inflight="3000" kube-apiserver I0318 01:19:00.304070 1154438 flags.go:64] FLAG: --min-request-timeout="3600" kube-apiserver I0318 01:19:00.304076 1154438 flags.go:64] FLAG: --oidc-ca-file="" kube-apiserver I0318 01:19:00.304081 1154438 flags.go:64] FLAG: --oidc-client-id="" kube-apiserver I0318 01:19:00.304086 1154438 flags.go:64] FLAG: --oidc-groups-claim="" kube-apiserver I0318 01:19:00.304090 1154438 flags.go:64] FLAG: --oidc-groups-prefix="" kube-apiserver I0318 01:19:00.304095 1154438 flags.go:64] FLAG: --oidc-issuer-url="" kube-apiserver I0318 01:19:00.304100 1154438 flags.go:64] FLAG: --oidc-required-claim="" kube-apiserver I0318 01:19:00.304108 1154438 flags.go:64] FLAG: --oidc-signing-algs="[RS256]" kube-apiserver I0318 01:19:00.304118 1154438 flags.go:64] FLAG: --oidc-username-claim="sub" kube-apiserver I0318 01:19:00.304123 1154438 flags.go:64] FLAG: --oidc-username-prefix="" kube-apiserver I0318 01:19:00.304128 1154438 flags.go:64] FLAG: --one_output="false" kube-apiserver I0318 01:19:00.304133 1154438 flags.go:64] FLAG: --openshift-config="/tmp/kube-apiserver-config-3180920900.yaml" kube-apiserver I0318 01:19:00.304140 1154438 flags.go:64] FLAG: --permit-address-sharing="false" kube-apiserver I0318 01:19:00.304145 1154438 flags.go:64] FLAG: --permit-port-sharing="false" kube-apiserver I0318 01:19:00.304150 1154438 flags.go:64] FLAG: --profiling="true" kube-apiserver I0318 01:19:00.304155 1154438 flags.go:64] FLAG: --proxy-client-cert-file="/var/lib/microshift/certs/aggregator-signer/aggregator-client/client.crt" kube-apiserver I0318 01:19:00.304161 1154438 flags.go:64] FLAG: --proxy-client-key-file="/var/lib/microshift/certs/aggregator-signer/aggregator-client/client.key" kube-apiserver I0318 01:19:00.304168 1154438 flags.go:64] FLAG: --request-timeout="1m0s" kube-apiserver I0318 01:19:00.304174 1154438 flags.go:64] FLAG: --requestheader-allowed-names="[kube-apiserver-proxy,system:kube-apiserver-proxy,system:openshift-aggregator]" kube-apiserver I0318 01:19:00.304183 1154438 flags.go:64] FLAG: --requestheader-client-ca-file="/var/lib/microshift/certs/aggregator-signer/ca.crt" kube-apiserver I0318 01:19:00.304189 1154438 flags.go:64] FLAG: --requestheader-extra-headers-prefix="[X-Remote-Extra-]" kube-apiserver I0318 01:19:00.304198 1154438 flags.go:64] FLAG: --requestheader-group-headers="[X-Remote-Group]" kube-apiserver I0318 01:19:00.304205 1154438 flags.go:64] FLAG: --requestheader-username-headers="[X-Remote-User]" kube-apiserver I0318 01:19:00.304213 1154438 flags.go:64] FLAG: --runtime-config="" kube-apiserver I0318 01:19:00.304219 1154438 flags.go:64] FLAG: --secure-port="6443" kube-apiserver I0318 01:19:00.304226 1154438 flags.go:64] FLAG: --send-retry-after-while-not-ready-once="true" kube-apiserver I0318 01:19:00.304231 1154438 flags.go:64] FLAG: --service-account-extend-token-expiration="true" kube-apiserver I0318 01:19:00.304236 1154438 flags.go:64] FLAG: --service-account-issuer="[https://kubernetes.default.svc]" kube-apiserver I0318 01:19:00.304245 1154438 flags.go:64] FLAG: --service-account-jwks-uri="" kube-apiserver I0318 01:19:00.304249 1154438 flags.go:64] FLAG: --service-account-key-file="[/var/lib/microshift/resources/kube-apiserver/secrets/service-account-key/service-account.pub]" kube-apiserver I0318 01:19:00.304270 1154438 flags.go:64] FLAG: --service-account-lookup="true" kube-apiserver I0318 01:19:00.304275 1154438 flags.go:64] FLAG: --service-account-max-token-expiration="0s" kube-apiserver I0318 01:19:00.304280 1154438 flags.go:64] FLAG: --service-account-signing-key-file="/var/lib/microshift/resources/kube-apiserver/secrets/service-account-key/service-account.key" kube-apiserver I0318 01:19:00.304290 1154438 flags.go:64] FLAG: --service-cluster-ip-range="10.43.0.0/16" kube-apiserver I0318 01:19:00.304295 1154438 flags.go:64] FLAG: --service-node-port-range="30000-32767" kube-apiserver I0318 01:19:00.304303 1154438 flags.go:64] FLAG: --show-hidden-metrics-for-version="" kube-apiserver I0318 01:19:00.304308 1154438 flags.go:64] FLAG: --shutdown-delay-duration="1m10s" kube-apiserver I0318 01:19:00.304313 1154438 flags.go:64] FLAG: --shutdown-send-retry-after="true" kube-apiserver I0318 01:19:00.304318 1154438 flags.go:64] FLAG: --skip_headers="false" kube-apiserver I0318 01:19:00.304323 1154438 flags.go:64] FLAG: --skip_log_headers="false" kube-apiserver I0318 01:19:00.304328 1154438 flags.go:64] FLAG: --stderrthreshold="2" kube-apiserver I0318 01:19:00.304334 1154438 flags.go:64] FLAG: --storage-backend="etcd3" kube-apiserver I0318 01:19:00.304339 1154438 flags.go:64] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf" kube-apiserver I0318 01:19:00.304345 1154438 flags.go:64] FLAG: --strict-transport-security-directives="[max-age=31536000,includeSubDomains,preload]" kube-apiserver I0318 01:19:00.304354 1154438 flags.go:64] FLAG: --tls-cert-file="/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt" kube-apiserver I0318 01:19:00.304362 1154438 flags.go:64] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]" kube-apiserver I0318 01:19:00.304377 1154438 flags.go:64] FLAG: --tls-min-version="VersionTLS12" kube-apiserver I0318 01:19:00.304383 1154438 flags.go:64] FLAG: --tls-private-key-file="/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key" kube-apiserver I0318 01:19:00.304391 1154438 flags.go:64] FLAG: --tls-sni-cert-key="[/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key]" kube-apiserver I0318 01:19:00.304414 1154438 flags.go:64] FLAG: --token-auth-file="" kube-apiserver I0318 01:19:00.304420 1154438 flags.go:64] FLAG: --tracing-config-file="" kube-apiserver I0318 01:19:00.304425 1154438 flags.go:64] FLAG: --v="2" kube-apiserver I0318 01:19:00.304431 1154438 flags.go:64] FLAG: --version="false" kube-apiserver I0318 01:19:00.304438 1154438 flags.go:64] FLAG: --vmodule="" kube-apiserver I0318 01:19:00.304443 1154438 flags.go:64] FLAG: --watch-cache="true" kube-apiserver I0318 01:19:00.304448 1154438 flags.go:64] FLAG: --watch-cache-sizes="[]" kube-apiserver I0318 01:19:00.304489 1154438 server.go:620] external host was not specified, using 10.43.0.1 kube-apiserver I0318 01:19:00.304900 1154438 server.go:201] Version: v1.25.0 kube-apiserver I0318 01:19:00.304918 1154438 server.go:203] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" kube-apiserver I0318 01:19:00.305494 1154438 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt::/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key" kube-apiserver I0318 01:19:00.306161 1154438 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt::/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key" kube-apiserver I0318 01:19:00.306682 1154438 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt::/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key" kube-apiserver I0318 01:19:00.307076 1154438 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt::/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key" kube-apiserver I0318 01:19:00.823189 1154438 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/var/lib/microshift/certs/ca-bundle/client-ca.crt" kube-apiserver I0318 01:19:00.823267 1154438 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="request-header::/var/lib/microshift/certs/aggregator-signer/ca.crt" kube-apiserver I0318 01:19:00.823648 1154438 shared_informer.go:255] Waiting for caches to sync for node_authorizer kube-apiserver I0318 01:19:00.824003 1154438 audit.go:350] Using audit backend: ignoreErrors Error: failed to initialize admission: failed to initialize admission plugin "PodSecurity": PodSecurity configuration error: [enforce: must be one of privileged, baseline, restricted, enforce-version: must be "latest" or "v1.x", audit: must be one of privileged, baseline, restricted, audit-version: must be "latest" or "v1.x", warn: must be one of privileged, baseline, restricted, warn-version: must be "latest" or "v1.x"] kube-apiserver E0318 01:19:00.825722 1154438 manager.go:116] service kube-apiserver exited with error: failed to initialize admission: failed to initialize admission plugin "PodSecurity": PodSecurity configuration error: [enforce: must be one of privileged, baseline, restricted, enforce-version: must be "latest" or "v1.x", audit: must be one of privileged, baseline, restricted, audit-version: must be "latest" or "v1.x", warn: must be one of privileged, baseline, restricted, warn-version: must be "latest" or "v1.x"], stopping MicroShift ??? I0318 01:19:00.826090 1154438 run.go:154] Interrupt received. Stopping services ??? E0318 01:19:00.826168 1154438 run.go:129] Stopped service-manager: context canceled {"level":"info","ts":"2023-03-18T01:19:00.826+0530","caller":"etcdserver/server.go:1462","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"8c1bff2098e6483a","current-leader-member-id":"8c1bff2098e6483a"} ??? I0318 01:19:00.826215 1154438 run.go:164] MicroShift stopped [root@microshift ~]#

dhellmann commented 1 year ago

@skylabreddy , your issue should be fixed by https://github.com/openshift/microshift/pull/1505 which will be in a z-stream release soon (I don't know if it made the deadline for this week, so it might be in next week's build).

skylabreddy commented 1 year ago

@dhellmann Is there any Tentative dates to fix the issue? is there any option to install old version of MicroShift?

skylabreddy commented 1 year ago

@dhellmann I see still same issue with current release also, microshift-4.1.2.8.

Service start issue.

Thanks Skylab

skylabreddy commented 1 year ago

Mar 21 19:41:20 microshift microshift[3888540]: kube-apiserver E0321 19:41:20.616357 3888540 manager.go:116] service kube-apiserver exited with error: failed to initialize admission: failed to initialize admission plugin "PodSecurity": PodSecurity configuration error: [enforce: must be one of privileged, baseline, restricted, enforce-version: must be "latest" or "v1.x", audit: must be one of privileged, baseline, restricted, audit-version: must be "latest" or "v1.x", warn: must be one of privileged, baseline, restricted, warn-version: must be "latest" or "v1.x"], stopping MicroShift Mar 21 19:41:23 microshift microshift[3888643]: Error: failed to initialize admission: failed to initialize admission plugin "PodSecurity": PodSecurity configuration error: [enforce: must be one of privileged, baseline, restricted, enforce-version: must be "latest" or "v1.x", audit: must be one of privileged, baseline, restricted, audit-version: must be "latest" or "v1.x", warn: must be one of privileged, baseline, restricted, warn-version: must be "latest" or "v1.x"]

dhellmann commented 1 year ago

I think we missed the build pipeline this week by a couple of hours, so it should be in the next build.

openshift / microshift