sealerio / sealer

Build, Share and Run Both Your Kubernetes Cluster and Distributed Applications (Project under CNCF)
http://sealer.cool
Apache License 2.0
2.06k stars 362 forks source link

sealer run kubernetes:v1.20.15-test success but cluster not work #1768

Closed jsparter closed 1 year ago

jsparter commented 2 years ago

What happen?

I run sealer run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.20.15-test as what #1763 said, and sealer didn't report error. But node is NotReady, and there is no pod in kube-system

[root@k8s-master-xx ~]# kubectl get no
NAME            STATUS     ROLES                  AGE   VERSION
k8s-master-xx   NotReady   control-plane,master   17m   v1.20.15
[root@k8s-master-xx ~]# kubectl get all -A
NAMESPACE         NAME                                   READY   STATUS             RESTARTS   AGE
tigera-operator   pod/tigera-operator-558ccbcb84-kw5m7   0/1     CrashLoopBackOff   5          17m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  17m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   17m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   0         0         0       0            0           kubernetes.io/os=linux   17m

NAMESPACE         NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
kube-system       deployment.apps/coredns           0/2     0            0           17m
tigera-operator   deployment.apps/tigera-operator   0/1     1            0           17m

NAMESPACE         NAME                                         DESIRED   CURRENT   READY   AGE
kube-system       replicaset.apps/coredns-86dfcb4f6f           2         0         0       17m
tigera-operator   replicaset.apps/tigera-operator-558ccbcb84   1         1         0       17m

I noticed there are containers:

[root@k8s-master-xxx ~]# docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                     PORTS               NAMES
6b8e6a9c350c        52468087127e             "operator"               4 minutes ago       Exited (1) 2 minutes ago                       k8s_tigera-operator_tigera-operator-558ccbcb84-kw5m7_tigera-operator_f7f6f86b-b06f-40c3-a592-69cfde3eb4e5_5
3497aa205c57        sea.hub:5000/pause:3.2   "/pause"                 19 minutes ago      Up 19 minutes                                  k8s_POD_tigera-operator-558ccbcb84-kw5m7_tigera-operator_f7f6f86b-b06f-40c3-a592-69cfde3eb4e5_0
03b7120e5ab7        0369cf4303ff             "etcd --advertise-cl…"   19 minutes ago      Up 19 minutes                                  k8s_etcd_etcd-k8s-master-32_kube-system_b5968553636a25d26d0b4e4bbd25e688_0
30ef77cd7034        323f6347f5e2             "kube-apiserver --ad…"   19 minutes ago      Up 19 minutes                                  k8s_kube-apiserver_kube-apiserver-k8s-master-32_kube-system_a0487ceb0b1e32c438ccce1e98c59882_0
c9d7c4bfe48f        9155e4deabb3             "kube-scheduler --ad…"   19 minutes ago      Up 19 minutes                                  k8s_kube-scheduler_kube-scheduler-k8s-master-32_kube-system_5be81472e5ae6e5467d525de0ebc71b5_0
5696753da03c        d6296d0e06d2             "kube-controller-man…"   19 minutes ago      Up 19 minutes                                  k8s_kube-controller-manager_kube-controller-manager-k8s-master-32_kube-system_75e9b4562c4b5cfe21a59c4f3713bf92_0
032cfec8ff11        sea.hub:5000/pause:3.2   "/pause"                 19 minutes ago      Up 19 minutes                                  k8s_POD_kube-scheduler-k8s-master-32_kube-system_5be81472e5ae6e5467d525de0ebc71b5_0
ed91dd39a7da        sea.hub:5000/pause:3.2   "/pause"                 19 minutes ago      Up 19 minutes                                  k8s_POD_kube-controller-manager-k8s-master-32_kube-system_75e9b4562c4b5cfe21a59c4f3713bf92_0
257a9db8b763        sea.hub:5000/pause:3.2   "/pause"                 19 minutes ago      Up 19 minutes                                  k8s_POD_kube-apiserver-k8s-master-32_kube-system_a0487ceb0b1e32c438ccce1e98c59882_0
ce5f4d24523b        sea.hub:5000/pause:3.2   "/pause"                 19 minutes ago      Up 19 minutes                                  k8s_POD_etcd-k8s-master-32_kube-system_b5968553636a25d26d0b4e4bbd25e688_0
67c60bbef923        registry:2.7.1           "registry serve /etc…"   20 minutes ago      Up 20 minutes                                  sealer-registry

Error logs of container 6b8e6a9c350c as follow:

E1012 07:53:46.154472       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167: Failed to watch *v1.BGPConfiguration: failed to list *v1.BGPConfiguration: bgpconfigurations.crd.projectcalico.org is forbidden: User "system:serviceaccount:tigera-operator:tigera-operator" cannot list resource "bgpconfigurations" in API group "crd.projectcalico.org" at the cluster scope
E1012 07:54:28.101046       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167: Failed to watch *v1.BGPConfiguration: failed to list *v1.BGPConfiguration: bgpconfigurations.crd.projectcalico.org is forbidden: User "system:serviceaccount:tigera-operator:tigera-operator" cannot list resource "bgpconfigurations" in API group "crd.projectcalico.org" at the cluster scope
{"level":"error","ts":1665561306.0906863,"logger":"controller.apiserver-controller","msg":"Could not wait for Cache to sync","error":"failed to wait for apiserver-controller caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/manager/internal.go:696"}
{"level":"error","ts":1665561306.0917573,"logger":"controller.tigera-installation-controller","msg":"Could not wait for Cache to sync","error":"failed to wait for tigera-installation-controller caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/manager/internal.go:696"}
{"level":"error","ts":1665561306.091953,"msg":"error received after stop sequence was engaged","error":"failed to wait for tigera-installation-controller caches to sync: timed out waiting for cache to be synced"}
{"level":"error","ts":1665561306.0919867,"msg":"error received after stop sequence was engaged","error":"leader election lost"}
{"level":"error","ts":1665561306.0921135,"logger":"setup","msg":"problem running manager","error":"failed to wait for apiserver-controller caches to sync: timed out waiting for cache to be synced"}

There is a deployment of coredns:

NAMESPACE         NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
kube-system       deployment.apps/coredns           0/2     0            0           17m

I'm confused why coredns isn't running, and whether it cause cluster not work?

Relevant log output?

output of sealer run:


[root@k8s-master-xx ~]# sealer run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.20.15-test -m xx.xx.xxx.xx
93a4ec00160fd76e5da279eb4bc787cdf716e8112ab10386e3cf62e25625610f
+ image_dir=/var/lib/sealer/data/my-cluster/rootfs/scripts/../images
···
+ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 21
 Server Version: 19.03.14-sealer
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ea765aba0d05254012b0b9e595e995c09186427f
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.17.1-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.772GiB
 Name: k8s-master-xx
 ID: FXHB:4RFC:B43I:TMBY:T4XP:7YPR:RKOL:O2RX:2E4M:N4CS:5GJF:LHD7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

+ load_images
···
2022-10-12 15:03:37 [INFO] [kube_certs.go:307] API server altNames: {map[apiserver.cluster.local:apiserver.cluster.local k8s                             -master-32:k8s-master-32 kubernetes:kubernetes kubernetes.default:kubernetes.default kubernetes.default.svc:kubernetes.defau                             lt.svc kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local localhost:localhost] map[xx.xx.xxx.xx:xx.xx.xxx.xx 10.103.97.2:10.103.97.2 10.96.0.1:10.96.0.1 xxxxxxxxxxxx 127.0.0.1:127.0.0.1 ::1:::1]}

2022-10-12 15:03:37 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "admin.conf" kubeconfig file

2022-10-12 15:03:37 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "controller-manager.conf" kubeconfig file

2022-10-12 15:03:37 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "scheduler.conf" kubeconfig file

2022-10-12 15:03:37 [INFO] [kubeconfig.go:266] [kubeconfig] Writing "kubelet.conf" kubeconfig file

+ disable_firewalld
···
# set by ack-distro
···
* Applying /etc/sysctl.d/99-sysctl.conf ...
···
+ systemctl enable kubelet
2022-10-12 15:03:37 [INFO] [init.go:155] start to init master0...

2022-10-12 15:03:38 [INFO] [init.go:182] join command is: kubeadm join  apiserver.cluster.local:6443 --token clegla.86r53vde                             jt9cm12w \
    --discovery-token-ca-cert-hash sha256:0b2080327d677ea0e09d261ec3e9024804046246d826b2f11b457b1ed2a09a09 \
    --control-plane --certificate-key 6aeae2959d81436531013433871284fd2e5230f117d77af14607785c50328707

2022-10-12 15:03:38 [INFO] [runtime.go:116] Succeeded in creating a new cluster, enjoy it!

customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
namespace/tigera-operator created
podsecuritypolicy.policy/tigera-operator created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
53f0cbaf126916fd6051e5c211e9fec8121024f8956a503298957dbce3c205bd

Error logs of tigera-operatoras container follow:

E1012 07:53:46.154472       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167: Failed to watch *v1.BGPConfiguration: failed to list *v1.BGPConfiguration: bgpconfigurations.crd.projectcalico.org is forbidden: User "system:serviceaccount:tigera-operator:tigera-operator" cannot list resource "bgpconfigurations" in API group "crd.projectcalico.org" at the cluster scope
E1012 07:54:28.101046       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167: Failed to watch *v1.BGPConfiguration: failed to list *v1.BGPConfiguration: bgpconfigurations.crd.projectcalico.org is forbidden: User "system:serviceaccount:tigera-operator:tigera-operator" cannot list resource "bgpconfigurations" in API group "crd.projectcalico.org" at the cluster scope
{"level":"error","ts":1665561306.0906863,"logger":"controller.apiserver-controller","msg":"Could not wait for Cache to sync","error":"failed to wait for apiserver-controller caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/manager/internal.go:696"}
{"level":"error","ts":1665561306.0917573,"logger":"controller.tigera-installation-controller","msg":"Could not wait for Cache to sync","error":"failed to wait for tigera-installation-controller caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/manager/internal.go:696"}
{"level":"error","ts":1665561306.091953,"msg":"error received after stop sequence was engaged","error":"failed to wait for tigera-installation-controller caches to sync: timed out waiting for cache to be synced"}
{"level":"error","ts":1665561306.0919867,"msg":"error received after stop sequence was engaged","error":"leader election lost"}
{"level":"error","ts":1665561306.0921135,"logger":"setup","msg":"problem running manager","error":"failed to wait for apiserver-controller caches to sync: timed out waiting for cache to be synced"}

Some logs of etcd container:

[root@k8s-master-32 ~]# docker logs -f 03b7120e5ab7
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2022-10-12 07:37:59.808765 I | etcdmain: etcd Version: 3.4.13
2022-10-12 07:37:59.808852 I | etcdmain: Git SHA: ae9734ed2
2022-10-12 07:37:59.808863 I | etcdmain: Go Version: go1.12.17
2022-10-12 07:37:59.808873 I | etcdmain: Go OS/Arch: linux/amd64
2022-10-12 07:37:59.808884 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2022-10-12 07:37:59.809130 I | embed: peerTLS: cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file =
2022-10-12 07:37:59.813810 I | embed: name = k8s-master-xx
2022-10-12 07:37:59.813860 I | embed: data dir = /var/lib/etcd
2022-10-12 07:37:59.813884 I | embed: member dir = /var/lib/etcd/member
2022-10-12 07:37:59.813938 I | embed: heartbeat = 100ms
2022-10-12 07:37:59.813958 I | embed: election = 1000ms
2022-10-12 07:37:59.813968 I | embed: snapshot count = 10000
2022-10-12 07:37:59.813992 I | embed: advertise client URLs = https://xx.xx.xxx.xx:2379
2022-10-12 07:37:59.835252 I | etcdserver: starting member fb0e721c0902c955 in cluster 3ff442b65d352e96
raft2022/10/12 07:37:59 INFO: fb0e721c0902c955 switched to configuration voters=()
raft2022/10/12 07:37:59 INFO: fb0e721c0902c955 became follower at term 0
raft2022/10/12 07:37:59 INFO: newRaft fb0e721c0902c955 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2022/10/12 07:37:59 INFO: fb0e721c0902c955 became follower at term 1
raft2022/10/12 07:37:59 INFO: fb0e721c0902c955 switched to configuration voters=(18090522217929689429)
2022-10-12 07:37:59.838139 W | auth: simple token is not cryptographically signed
2022-10-12 07:37:59.849918 I | etcdserver: starting server... [version: 3.4.13, cluster version: to_be_decided]
2022-10-12 07:37:59.852807 I | etcdserver: fb0e721c0902c955 as single-node; fast-forwarding 9 ticks (election ticks 10)
raft2022/10/12 07:37:59 INFO: fb0e721c0902c955 switched to configuration voters=(18090522217929689429)
2022-10-12 07:37:59.856996 I | etcdserver/membership: added member fb0e721c0902c955 [https://10.10.101.32:2380] to cluster 3ff442b65d352e96
2022-10-12 07:37:59.860546 I | embed: ClientTLS: cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file =
2022-10-12 07:37:59.861316 I | embed: listening for metrics on http://0.0.0.0:2381
2022-10-12 07:37:59.861566 I | embed: listening for peers on xx.xx.xxx.xx:2380
raft2022/10/12 07:38:00 INFO: fb0e721c0902c955 is starting a new election at term 1
raft2022/10/12 07:38:00 INFO: fb0e721c0902c955 became candidate at term 2
raft2022/10/12 07:38:00 INFO: fb0e721c0902c955 received MsgVoteResp from fb0e721c0902c955 at term 2
raft2022/10/12 07:38:00 INFO: fb0e721c0902c955 became leader at term 2
raft2022/10/12 07:38:00 INFO: raft.node: fb0e721c0902c955 elected leader fb0e721c0902c955 at term 2
2022-10-12 07:38:00.040257 I | etcdserver: setting up the initial cluster version to 3.4
2022-10-12 07:38:00.042315 N | etcdserver/membership: set the initial cluster version to 3.4
2022-10-12 07:38:00.042555 I | etcdserver/api: enabled capabilities for version 3.4
2022-10-12 07:38:00.042662 I | etcdserver: published {Name:k8s-master-32 ClientURLs:[https://10.10.101.32:2379]} to cluster 3ff442b65d352e96
2022-10-12 07:38:00.043648 I | embed: ready to serve client requests
2022-10-12 07:38:00.045547 I | embed: ready to serve client requests
2022-10-12 07:38:00.049823 I | embed: serving client requests on xx.xx.xxx.xx:2379
2022-10-12 07:38:00.053566 I | embed: serving client requests on 127.0.0.1:2379
2022-10-12 07:38:23.188645 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "range_response_count:1 size:420" took too long (270.622163ms) to execute
2022-10-12 07:38:28.512699 W | etcdserver/api/etcdhttp: /health error; QGET failed etcdserver: request timed out (status code 503)
2022-10-12 07:38:28.515693 W | wal: sync duration of 1.75577417s, expected less than 1s
2022-10-12 07:38:28.622703 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:489" took too long (1.694819186s) to execute
2022-10-12 07:38:28.623125 W | etcdserver: request "header:<ID:14507646683098849265 > lease_revoke:<id:495583cb20fb1e4f>" with result "size:28" took too long (106.64124ms) to execute
2022-10-12 07:38:28.623956 W | etcdserver: read-only range request "key:\"/registry/serviceaccounts/kube-system/node-controller\" " with result "range_response_count:1 size:242" took too long (1.691472226s) to execute
2022-10-12 07:38:28.624189 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:5" took too long (601.18423ms) to execute
2022-10-12 07:38:37.513856 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2022-10-12 08:05:31.942537 W | etcdserver: request "header:<ID:14507646683098861153 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:3539 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:441 >> failure:<request_range:<key:\"/registry/leases/kube-system/kube-controller-manager\" > >>" with result "size:16" took too long (386.806927ms) to execute

Some logs of apiserver:

W1012 15:38:04.705755       1 genericapiserver.go:425] Skipping API batch/v2alpha1 because it has no resources.
W1012 15:38:04.752292       1 genericapiserver.go:425] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W1012 15:38:04.792462       1 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W1012 15:38:04.828482       1 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W1012 15:38:04.849886       1 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W1012 15:38:04.878260       1 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W1012 15:38:04.888204       1 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W1012 15:38:04.910303       1 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
W1012 15:38:04.910415       1 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
I1012 15:38:10.073795       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I1012 15:38:10.074081       1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
E1012 15:38:10.161120       1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/10.10.101.32, ResourceVersion: 0, AdditionalErrorMsg:
E1012 15:38:21.351620       1 customresource_handler.go:669] error building openapi models for installations.operator.tigera.io: ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.spec.properties.componentResources.items.<array>.properties.resourceRequirements.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.spec.properties.componentResources.items.<array>.properties.resourceRequirements.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.spec.properties.nodeUpdateStrategy.properties.rollingUpdate.properties.maxSurge has invalid property: anyOf
ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.spec.properties.nodeUpdateStrategy.properties.rollingUpdate.properties.maxUnavailable has invalid property: anyOf
ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.status.properties.computed.properties.componentResources.items.<array>.properties.resourceRequirements.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.status.properties.computed.properties.componentResources.items.<array>.properties.resourceRequirements.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.status.properties.computed.properties.nodeUpdateStrategy.properties.rollingUpdate.properties.maxSurge has invalid property: anyOf
ERROR $root.definitions.io.tigera.operator.v1.Installation.properties.status.properties.computed.properties.nodeUpdateStrategy.properties.rollingUpdate.properties.maxUnavailable has invalid property: anyOf

What you expected to happen?

sealer run successfully and cluster works properly

How to reproduce it (as minimally and precisely as possible)?

sealer run registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes:v1.20.15-test -m xx.xx.xxx.xx

Anything else we need to know?

No response

What is the version of Sealer you using?

{"gitVersion":"unknown","gitCommit":"50b1c7aa","buildDate":"2022-10-12 12:00:38","goVersion":"go1.17.7","compiler":"gc","platform":"linux/amd64"}

What is your OS environment?

CentOS Linux 7

What is the Kernel version?

Linux 5.17.1-1.el7.elrepo.x86_64

Other environment you want to tell us?

kakaZhou719 commented 2 years ago

@jsparter ,could you pls to check the content kubeadm.yaml cat /etc/kubernetes/kubeadm.yaml | grep PodSecurityPolicy

jsparter commented 2 years ago

@kakaZhou719

[root@k8s-master-xx ~]# cat /etc/kubernetes/kubeadm.yaml | grep PodSecurityPolicy
    enable-admission-plugins: PodSecurityPolicy,NodeRestriction
kakaZhou719 commented 2 years ago

oh, if we set PodSecurityPolicy in ClusterConfiguration at kubeadm.yaml, this admission plugin will restrict the pod creation. @Stevent-fei already update this cluster image, pls update you cluster image and try again.

tgfree7 commented 2 years ago

oh, if we set PodSecurityPolicy in ClusterConfiguration at kubeadm.yaml, this admission plugin will restrict the pod creation. @Stevent-fei already update this cluster image, pls update you cluster image and try again.

@kakaZhou719 pull the same image, and can not make calico running. there is operator's log

E1013 02:44:35.832437       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167: Failed to watch *v1.BGPConfiguration: failed to list *v1.BGPConfiguration: bgpconfigurations.crd.projectcalico.org is forbidden: User "system:serviceaccount:tigera-operator:tigera-operator" cannot list resource "bgpconfigurations" in API group "crd.projectcalico.org" at the cluster scope
E1013 02:44:58.665221       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.3/tools/cache/reflector.go:167: Failed to watch *v1.BGPConfiguration: failed to list *v1.BGPConfiguration: bgpconfigurations.crd.projectcalico.org is forbidden: User "system:serviceaccount:tigera-operator:tigera-operator" cannot list resource "bgpconfigurations" in API group "crd.projectcalico.org" at the cluster scope

and if edit the clusterrole for bgpconfigurations, still not working:

{"level":"error","ts":1665629324.9539669,"logger":"controller.tigera-installation-controller","msg":"Reconciler error","name":"tigera-operator-token-frnml","namespace":"tigera-operator","error":"Could not resolve CalicoNetwork IPPool and kubeadm configuration: IPPool 100.64.0.0/10 is not within the platform's configured pod network CIDR(s) [100.64.0.0/16]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:214"}
{"level":"error","ts":1665629325.0432544,"logger":"controller_installation","msg":"ResourceReadError","Request.Namespace":"","Request.Name":"calico","ResourceReadError":"Error querying installation","error":"Could not resolve CalicoNetwork IPPool and kubeadm configuration: IPPool 100.64.0.0/10 is not within the platform's configured pod network CIDR(s) [100.64.0.0/16]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:214"}
{"level":"error","ts":1665629325.043346,"logger":"controller.tigera-installation-controller","msg":"Reconciler error","name":"calico","namespace":"","error":"Could not resolve CalicoNetwork IPPool and kubeadm configuration: IPPool 100.64.0.0/10 is not within the platform's configured pod network CIDR(s) [100.64.0.0/16]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:214"}

FYI,image id is 93a4ec00160f

jsparter commented 2 years ago

oh, if we set PodSecurityPolicy in ClusterConfiguration at kubeadm.yaml, this admission plugin will restrict the pod creation. @Stevent-fei already update this cluster image, pls update you cluster image and try again.

@kakaZhou719 I pulled this image again, but I found that imageId dosen't change, and the problem still here

REPOSITORY                                              TAG             IMAGE ID       CREATED        SIZE
registry.cn-qingdao.aliyuncs.com/sealer-io/kubernetes   v1.20.15-test   93a4ec00160f   28 hours ago   865 MB
Stevent-fei commented 1 year ago

I've fixed it and can see the list of currently supported images in readme. If there is no problem, I will close this issue before December 2nd.