Closed gyliu513 closed 3 years ago
What steps did you take and what happened: [A clear and concise description on how to REPRODUCE the bug.]
root@xanthene1:~# kubectl logs -f cluster-sample-apiserver-0 I0720 08:12:01.462735 1 flags.go:33] FLAG: --add-dir-header="false" I0720 08:12:01.462790 1 flags.go:33] FLAG: --address="127.0.0.1" I0720 08:12:01.462797 1 flags.go:33] FLAG: --admission-control="[]" I0720 08:12:01.463760 1 flags.go:33] FLAG: --admission-control-config-file="" I0720 08:12:01.463769 1 flags.go:33] FLAG: --advertise-address="<nil>" I0720 08:12:01.463774 1 flags.go:33] FLAG: --allow-privileged="true" I0720 08:12:01.463780 1 flags.go:33] FLAG: --alsologtostderr="false" I0720 08:12:01.463785 1 flags.go:33] FLAG: --anonymous-auth="true" I0720 08:12:01.463795 1 flags.go:33] FLAG: --api-audiences="[]" I0720 08:12:01.463805 1 flags.go:33] FLAG: --apiserver-count="1" I0720 08:12:01.463810 1 flags.go:33] FLAG: --audit-dynamic-configuration="false" I0720 08:12:01.463814 1 flags.go:33] FLAG: --audit-log-batch-buffer-size="10000" I0720 08:12:01.463818 1 flags.go:33] FLAG: --audit-log-batch-max-size="1" I0720 08:12:01.463822 1 flags.go:33] FLAG: --audit-log-batch-max-wait="0s" I0720 08:12:01.463830 1 flags.go:33] FLAG: --audit-log-batch-throttle-burst="0" I0720 08:12:01.463834 1 flags.go:33] FLAG: --audit-log-batch-throttle-enable="false" I0720 08:12:01.463838 1 flags.go:33] FLAG: --audit-log-batch-throttle-qps="0" I0720 08:12:01.463843 1 flags.go:33] FLAG: --audit-log-format="json" I0720 08:12:01.463847 1 flags.go:33] FLAG: --audit-log-maxage="0" I0720 08:12:01.463854 1 flags.go:33] FLAG: --audit-log-maxbackup="0" I0720 08:12:01.463858 1 flags.go:33] FLAG: --audit-log-maxsize="0" I0720 08:12:01.463862 1 flags.go:33] FLAG: --audit-log-mode="blocking" I0720 08:12:01.463867 1 flags.go:33] FLAG: --audit-log-path="" I0720 08:12:01.463870 1 flags.go:33] FLAG: --audit-log-truncate-enabled="false" I0720 08:12:01.463874 1 flags.go:33] FLAG: --audit-log-truncate-max-batch-size="10485760" I0720 08:12:01.463884 1 flags.go:33] FLAG: --audit-log-truncate-max-event-size="102400" I0720 08:12:01.463888 1 flags.go:33] FLAG: --audit-log-version="audit.k8s.io/v1" I0720 08:12:01.463892 1 flags.go:33] FLAG: --audit-policy-file="" I0720 08:12:01.463895 1 flags.go:33] FLAG: --audit-webhook-batch-buffer-size="10000" I0720 08:12:01.463899 1 flags.go:33] FLAG: --audit-webhook-batch-initial-backoff="10s" I0720 08:12:01.463903 1 flags.go:33] FLAG: --audit-webhook-batch-max-size="400" I0720 08:12:01.463910 1 flags.go:33] FLAG: --audit-webhook-batch-max-wait="30s" I0720 08:12:01.463914 1 flags.go:33] FLAG: --audit-webhook-batch-throttle-burst="15" I0720 08:12:01.463917 1 flags.go:33] FLAG: --audit-webhook-batch-throttle-enable="true" I0720 08:12:01.463921 1 flags.go:33] FLAG: --audit-webhook-batch-throttle-qps="10" I0720 08:12:01.463925 1 flags.go:33] FLAG: --audit-webhook-config-file="" I0720 08:12:01.463933 1 flags.go:33] FLAG: --audit-webhook-initial-backoff="10s" I0720 08:12:01.463936 1 flags.go:33] FLAG: --audit-webhook-mode="batch" I0720 08:12:01.463940 1 flags.go:33] FLAG: --audit-webhook-truncate-enabled="false" I0720 08:12:01.463944 1 flags.go:33] FLAG: --audit-webhook-truncate-max-batch-size="10485760" I0720 08:12:01.463948 1 flags.go:33] FLAG: --audit-webhook-truncate-max-event-size="102400" I0720 08:12:01.463952 1 flags.go:33] FLAG: --audit-webhook-version="audit.k8s.io/v1" I0720 08:12:01.463960 1 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s" I0720 08:12:01.463967 1 flags.go:33] FLAG: --authentication-token-webhook-config-file="" I0720 08:12:01.463971 1 flags.go:33] FLAG: --authorization-mode="[Node,RBAC]" I0720 08:12:01.463990 1 flags.go:33] FLAG: --authorization-policy-file="" I0720 08:12:01.463993 1 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" I0720 08:12:01.463997 1 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" I0720 08:12:01.464028 1 flags.go:33] FLAG: --authorization-webhook-config-file="" I0720 08:12:01.464032 1 flags.go:33] FLAG: --basic-auth-file="" I0720 08:12:01.464036 1 flags.go:33] FLAG: --bind-address="0.0.0.0" I0720 08:12:01.464040 1 flags.go:33] FLAG: --cert-dir="/var/run/kubernetes" I0720 08:12:01.464044 1 flags.go:33] FLAG: --client-ca-file="/etc/kubernetes/pki/apiserver/ca/tls.crt" I0720 08:12:01.464049 1 flags.go:33] FLAG: --cloud-config="" I0720 08:12:01.464056 1 flags.go:33] FLAG: --cloud-provider="" I0720 08:12:01.464059 1 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16" I0720 08:12:01.464071 1 flags.go:33] FLAG: --contention-profiling="false" I0720 08:12:01.464074 1 flags.go:33] FLAG: --cors-allowed-origins="[]" I0720 08:12:01.464081 1 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300" I0720 08:12:01.464090 1 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300" I0720 08:12:01.464094 1 flags.go:33] FLAG: --default-watch-cache-size="100" I0720 08:12:01.464098 1 flags.go:33] FLAG: --delete-collection-workers="1" I0720 08:12:01.464101 1 flags.go:33] FLAG: --deserialization-cache-size="0" I0720 08:12:01.464105 1 flags.go:33] FLAG: --disable-admission-plugins="[]" I0720 08:12:01.464112 1 flags.go:33] FLAG: --egress-selector-config-file="" I0720 08:12:01.464119 1 flags.go:33] FLAG: --enable-admission-plugins="[NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota]" I0720 08:12:01.464148 1 flags.go:33] FLAG: --enable-aggregator-routing="true" I0720 08:12:01.464154 1 flags.go:33] FLAG: --enable-bootstrap-token-auth="true" I0720 08:12:01.464157 1 flags.go:33] FLAG: --enable-garbage-collector="true" I0720 08:12:01.464161 1 flags.go:33] FLAG: --enable-inflight-quota-handler="false" I0720 08:12:01.464169 1 flags.go:33] FLAG: --enable-logs-handler="true" I0720 08:12:01.464172 1 flags.go:33] FLAG: --enable-swagger-ui="false" I0720 08:12:01.464176 1 flags.go:33] FLAG: --encryption-provider-config="" I0720 08:12:01.464179 1 flags.go:33] FLAG: --endpoint-reconciler-type="master-count" I0720 08:12:01.464183 1 flags.go:33] FLAG: --etcd-cafile="/etc/kubernetes/pki/etcd/ca/tls.crt" I0720 08:12:01.464187 1 flags.go:33] FLAG: --etcd-certfile="/etc/kubernetes/pki/etcd/tls.crt" I0720 08:12:01.464195 1 flags.go:33] FLAG: --etcd-compaction-interval="5m0s" I0720 08:12:01.464198 1 flags.go:33] FLAG: --etcd-count-metric-poll-period="1m0s" I0720 08:12:01.464202 1 flags.go:33] FLAG: --etcd-keyfile="/etc/kubernetes/pki/etcd/tls.key" I0720 08:12:01.464206 1 flags.go:33] FLAG: --etcd-prefix="/registry" I0720 08:12:01.464210 1 flags.go:33] FLAG: --etcd-servers="[https://cluster-sample-etcd-0.cluster-sample-etcd.default:2379]" I0720 08:12:01.464220 1 flags.go:33] FLAG: --etcd-servers-overrides="[]" I0720 08:12:01.464234 1 flags.go:33] FLAG: --event-ttl="1h0m0s" I0720 08:12:01.464238 1 flags.go:33] FLAG: --experimental-encryption-provider-config="" I0720 08:12:01.464242 1 flags.go:33] FLAG: --external-hostname="" I0720 08:12:01.464245 1 flags.go:33] FLAG: --feature-gates="" I0720 08:12:01.464252 1 flags.go:33] FLAG: --help="false" I0720 08:12:01.464260 1 flags.go:33] FLAG: --http2-max-streams-per-connection="0" I0720 08:12:01.464263 1 flags.go:33] FLAG: --insecure-bind-address="127.0.0.1" I0720 08:12:01.464274 1 flags.go:33] FLAG: --insecure-port="8080" I0720 08:12:01.464284 1 flags.go:33] FLAG: --kubelet-certificate-authority="/etc/kubernetes/pki/apiserver/ca/tls.crt" I0720 08:12:01.464289 1 flags.go:33] FLAG: --kubelet-client-certificate="/etc/kubernetes/pki/kubelet/tls.crt" I0720 08:12:01.464293 1 flags.go:33] FLAG: --kubelet-client-key="/etc/kubernetes/pki/kubelet/tls.key" I0720 08:12:01.464301 1 flags.go:33] FLAG: --kubelet-https="true" I0720 08:12:01.464305 1 flags.go:33] FLAG: --kubelet-port="10250" I0720 08:12:01.464311 1 flags.go:33] FLAG: --kubelet-preferred-address-types="[InternalIP,ExternalIP]" I0720 08:12:01.464324 1 flags.go:33] FLAG: --kubelet-read-only-port="10255" I0720 08:12:01.464328 1 flags.go:33] FLAG: --kubelet-timeout="5s" I0720 08:12:01.464332 1 flags.go:33] FLAG: --kubernetes-service-node-port="0" I0720 08:12:01.464340 1 flags.go:33] FLAG: --livez-grace-period="0s" I0720 08:12:01.464344 1 flags.go:33] FLAG: --log-backtrace-at=":0" I0720 08:12:01.464351 1 flags.go:33] FLAG: --log-dir="" I0720 08:12:01.464355 1 flags.go:33] FLAG: --log-file="" I0720 08:12:01.464359 1 flags.go:33] FLAG: --log-file-max-size="1800" I0720 08:12:01.464371 1 flags.go:33] FLAG: --log-flush-frequency="5s" I0720 08:12:01.464375 1 flags.go:33] FLAG: --logtostderr="true" I0720 08:12:01.464378 1 flags.go:33] FLAG: --master-service-namespace="default" I0720 08:12:01.464383 1 flags.go:33] FLAG: --max-connection-bytes-per-sec="0" I0720 08:12:01.464387 1 flags.go:33] FLAG: --max-mutating-requests-inflight="200" I0720 08:12:01.464391 1 flags.go:33] FLAG: --max-requests-inflight="400" I0720 08:12:01.464401 1 flags.go:33] FLAG: --min-request-timeout="1800" I0720 08:12:01.464405 1 flags.go:33] FLAG: --oidc-ca-file="" I0720 08:12:01.464420 1 flags.go:33] FLAG: --oidc-client-id="" I0720 08:12:01.464428 1 flags.go:33] FLAG: --oidc-groups-claim="" I0720 08:12:01.464432 1 flags.go:33] FLAG: --oidc-groups-prefix="" I0720 08:12:01.464435 1 flags.go:33] FLAG: --oidc-issuer-url="" I0720 08:12:01.464445 1 flags.go:33] FLAG: --oidc-required-claim="" I0720 08:12:01.464466 1 flags.go:33] FLAG: --oidc-signing-algs="[RS256]" I0720 08:12:01.464478 1 flags.go:33] FLAG: --oidc-username-claim="sub" I0720 08:12:01.464482 1 flags.go:33] FLAG: --oidc-username-prefix="" I0720 08:12:01.464485 1 flags.go:33] FLAG: --port="8080" I0720 08:12:01.464493 1 flags.go:33] FLAG: --profiling="true" I0720 08:12:01.464497 1 flags.go:33] FLAG: --proxy-client-cert-file="/etc/kubernetes/pki/proxy/tls.crt" I0720 08:12:01.464501 1 flags.go:33] FLAG: --proxy-client-key-file="/etc/kubernetes/pki/proxy/tls.key" I0720 08:12:01.464506 1 flags.go:33] FLAG: --request-timeout="1m0s" I0720 08:12:01.464510 1 flags.go:33] FLAG: --requestheader-allowed-names="[front-proxy-client]" I0720 08:12:01.464525 1 flags.go:33] FLAG: --requestheader-client-ca-file="/etc/kubernetes/pki/root/tls.crt" I0720 08:12:01.464541 1 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[X-Remote-Extra-]" I0720 08:12:01.464559 1 flags.go:33] FLAG: --requestheader-group-headers="[X-Remote-Group]" I0720 08:12:01.464574 1 flags.go:33] FLAG: --requestheader-username-headers="[X-Remote-User]" I0720 08:12:01.464584 1 flags.go:33] FLAG: --runtime-config="api/all=" I0720 08:12:01.464596 1 flags.go:33] FLAG: --secure-port="6443" I0720 08:12:01.464609 1 flags.go:33] FLAG: --service-account-api-audiences="[]" I0720 08:12:01.464626 1 flags.go:33] FLAG: --service-account-issuer="" I0720 08:12:01.464634 1 flags.go:33] FLAG: --service-account-key-file="[/etc/kubernetes/pki/service-account/tls.key]" I0720 08:12:01.464662 1 flags.go:33] FLAG: --service-account-lookup="true" I0720 08:12:01.464674 1 flags.go:33] FLAG: --service-account-max-token-expiration="0s" I0720 08:12:01.464683 1 flags.go:33] FLAG: --service-account-signing-key-file="" I0720 08:12:01.464686 1 flags.go:33] FLAG: --service-cluster-ip-range="10.32.0.0/16" I0720 08:12:01.464690 1 flags.go:33] FLAG: --service-node-port-range="30000-32767" I0720 08:12:01.464704 1 flags.go:33] FLAG: --shutdown-delay-duration="0s" I0720 08:12:01.464708 1 flags.go:33] FLAG: --skip-headers="false" I0720 08:12:01.464712 1 flags.go:33] FLAG: --skip-log-headers="false" I0720 08:12:01.464720 1 flags.go:33] FLAG: --ssh-keyfile="" I0720 08:12:01.464724 1 flags.go:33] FLAG: --ssh-user="" I0720 08:12:01.464728 1 flags.go:33] FLAG: --stderrthreshold="2" I0720 08:12:01.464732 1 flags.go:33] FLAG: --storage-backend="" I0720 08:12:01.464736 1 flags.go:33] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf" I0720 08:12:01.464744 1 flags.go:33] FLAG: --target-ram-mb="0" I0720 08:12:01.464748 1 flags.go:33] FLAG: --tls-cert-file="/etc/kubernetes/pki/apiserver/tls.crt" I0720 08:12:01.464752 1 flags.go:33] FLAG: --tls-cipher-suites="[]" I0720 08:12:01.464765 1 flags.go:33] FLAG: --tls-min-version="" I0720 08:12:01.464769 1 flags.go:33] FLAG: --tls-private-key-file="/etc/kubernetes/pki/apiserver/tls.key" I0720 08:12:01.464774 1 flags.go:33] FLAG: --tls-sni-cert-key="[]" I0720 08:12:01.464787 1 flags.go:33] FLAG: --token-auth-file="" I0720 08:12:01.464791 1 flags.go:33] FLAG: --v="2" I0720 08:12:01.464795 1 flags.go:33] FLAG: --version="false" I0720 08:12:01.464801 1 flags.go:33] FLAG: --vmodule="" I0720 08:12:01.464818 1 flags.go:33] FLAG: --watch-cache="true" I0720 08:12:01.464828 1 flags.go:33] FLAG: --watch-cache-sizes="[]" I0720 08:12:01.465199 1 server.go:623] external host was not specified, using 10.244.0.22 I0720 08:12:01.465377 1 server.go:666] Initializing cache sizes based on 0MB limit I0720 08:12:01.465723 1 server.go:149] Version: v1.16.2 Error: unable to load client CA file: unable to load client CA file: open /etc/kubernetes/pki/root/tls.crt: no such file or directory Usage: kube-apiserver [flags] Generic flags: --advertise-address ip The IP address on which to advertise the apiserver to members of the cluster. This address must be reachable by the rest of the cluster. If blank, the --bind-address will be used. If --bind-address is unspecified, the host's default interface will be used. --cors-allowed-origins strings List of allowed origins for CORS, comma separated. An allowed origin can be a regular expression to support subdomain matching. If this list is empty CORS will not be enabled. --default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) --default-unreachable-toleration-seconds int Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) --enable-inflight-quota-handler If true, replace the max-in-flight handler with an enhanced one that queues and dispatches with priority and fairness --external-hostname string The hostname to use when generating externalized URLs for this master (e.g. Swagger API Docs). --feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are: APIListChunking=true|false (BETA - default=true) APIResponseCompression=true|false (BETA - default=true) AllAlpha=true|false (ALPHA - default=false) AppArmor=true|false (BETA - default=true) AttachVolumeLimit=true|false (BETA - default=true) BalanceAttachedNodeVolumes=true|false (ALPHA - default=false) BlockVolume=true|false (BETA - default=true) BoundServiceAccountTokenVolume=true|false (ALPHA - default=false) CPUManager=true|false (BETA - default=true) CRIContainerLogRotation=true|false (BETA - default=true) CSIBlockVolume=true|false (BETA - default=true) CSIDriverRegistry=true|false (BETA - default=true) CSIInlineVolume=true|false (BETA - default=true) CSIMigration=true|false (ALPHA - default=false) CSIMigrationAWS=true|false (ALPHA - default=false) CSIMigrationAzureDisk=true|false (ALPHA - default=false) CSIMigrationAzureFile=true|false (ALPHA - default=false) CSIMigrationGCE=true|false (ALPHA - default=false) CSIMigrationOpenStack=true|false (ALPHA - default=false) CSINodeInfo=true|false (BETA - default=true) CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false) CustomResourceDefaulting=true|false (BETA - default=true) DevicePlugins=true|false (BETA - default=true) DryRun=true|false (BETA - default=true) DynamicAuditing=true|false (ALPHA - default=false) DynamicKubeletConfig=true|false (BETA - default=true) EndpointSlice=true|false (ALPHA - default=false) EphemeralContainers=true|false (ALPHA - default=false) EvenPodsSpread=true|false (ALPHA - default=false) ExpandCSIVolumes=true|false (BETA - default=true) ExpandInUsePersistentVolumes=true|false (BETA - default=true) ExpandPersistentVolumes=true|false (BETA - default=true) ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false) HPAScaleToZero=true|false (ALPHA - default=false) HyperVContainer=true|false (ALPHA - default=false) IPv6DualStack=true|false (ALPHA - default=false) KubeletPodResources=true|false (BETA - default=true) LegacyNodeRoleBehavior=true|false (ALPHA - default=true) LocalStorageCapacityIsolation=true|false (BETA - default=true) LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false) MountContainers=true|false (ALPHA - default=false) NodeDisruptionExclusion=true|false (ALPHA - default=false) NodeLease=true|false (BETA - default=true) NonPreemptingPriority=true|false (ALPHA - default=false) PodOverhead=true|false (ALPHA - default=false) PodShareProcessNamespace=true|false (BETA - default=true) ProcMountType=true|false (ALPHA - default=false) QOSReserved=true|false (ALPHA - default=false) RemainingItemCount=true|false (BETA - default=true) RemoveSelfLink=true|false (ALPHA - default=false) RequestManagement=true|false (ALPHA - default=false) ResourceLimitsPriorityFunction=true|false (ALPHA - default=false) ResourceQuotaScopeSelectors=true|false (BETA - default=true) RotateKubeletClientCertificate=true|false (BETA - default=true) RotateKubeletServerCertificate=true|false (BETA - default=true) RunAsGroup=true|false (BETA - default=true) RuntimeClass=true|false (BETA - default=true) SCTPSupport=true|false (ALPHA - default=false) ScheduleDaemonSetPods=true|false (BETA - default=true) ServerSideApply=true|false (BETA - default=true) ServiceLoadBalancerFinalizer=true|false (BETA - default=true) ServiceNodeExclusion=true|false (ALPHA - default=false) StartupProbe=true|false (ALPHA - default=false) StorageVersionHash=true|false (BETA - default=true) StreamingProxyRedirects=true|false (BETA - default=true) SupportNodePidsLimit=true|false (BETA - default=true) SupportPodPidsLimit=true|false (BETA - default=true) Sysctls=true|false (BETA - default=true) TTLAfterFinished=true|false (ALPHA - default=false) TaintBasedEvictions=true|false (BETA - default=true) TaintNodesByCondition=true|false (BETA - default=true) TokenRequest=true|false (BETA - default=true) TokenRequestProjection=true|false (BETA - default=true) TopologyManager=true|false (ALPHA - default=false) ValidateProxyRedirects=true|false (BETA - default=true) VolumePVCDataSource=true|false (BETA - default=true) VolumeSnapshotDataSource=true|false (ALPHA - default=false) VolumeSubpathEnvExpansion=true|false (BETA - default=true) WatchBookmark=true|false (BETA - default=true) WinDSR=true|false (ALPHA - default=false) WinOverlay=true|false (ALPHA - default=false) WindowsGMSA=true|false (BETA - default=true) WindowsRunAsUserName=true|false (ALPHA - default=false) --livez-grace-period duration This option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has elapsed,
@wangjsty ^^
What did you expect to happen:
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version
/etc/os-release
/kind bug [One or more /area label. See https://github.com/kubernetes-sigs/cluster-api-provider-nested/labels?q=area for the list of labels]
What steps did you take and what happened: [A clear and concise description on how to REPRODUCE the bug.]
@wangjsty ^^
What did you expect to happen:
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version):/etc/os-release):/kind bug [One or more /area label. See https://github.com/kubernetes-sigs/cluster-api-provider-nested/labels?q=area for the list of labels]