Closed zhao-kun closed 3 years ago
There is a known bugs about EG deployment, which need to be discussed with @xxx7xxxx and @benja-wu for more details
member.yaml
contents in some node (not all node):clusterMembers:
- id: 1031933740400126356
name: ""
peerURL: http://easemesh-control-plane-2.easemesh-controlplane-hs.easemesh:2380
- id: 5797175598832725525
name: easemesh-control-plane-0
peerURL: http://easemesh-control-plane-0.easemesh-controlplane-hs.easemesh:2380
- id: 1664029883460182567
name: easemesh-control-plane-1
peerURL: http://easemesh-control-plane-1.easemesh-controlplane-hs.easemesh:2380
knownMembers:
- id: 1031933740400126356
name: ""
peerURL: http://easemesh-control-plane-2.easemesh-controlplane-hs.easemesh:2380
- id: 5797175598832725525
name: easemesh-control-plane-0
peerURL: http://easemesh-control-plane-0.easemesh-controlplane-hs.easemesh:2380
- id: 1664029883460182567
name: easemesh-control-plane-1
peerURL: http://easemesh-control-plane-1.easemesh-controlplane-hs.easemesh:2380
Once the situation appeared, no matter how many times I reboot EG, EG can't be boot normally.
Logs in leader:
cluster-join-urls [http://easemesh-control-plane-0.easemesh-controlplane-hs.easemesh:2380 http://easemesh-control-plane-1.easemesh-controlplane-hs.easemesh:2380 http://easemesh-control-plane-2.easemesh-controlplane-hs.easemesh:2380] changed to empty because it tries to join itself
2021-06-04T11:10:22.205Z INFO server/main.go:61 Easegress release: 1.0.0, repo: https://github.com/zhao-kun/easegress-1, commit: git-0122cf3
2021-06-04T11:10:22.205Z INFO storage/storage.go:250 /running_objects.yaml not exist
2021-06-04T11:10:22.205Z INFO cluster/cluster.go:382 client connect with endpoints: [http://easemesh-control-plane-2.easemesh-controlplane-hs.easemesh:2380 http://easemesh-control-plane-0.easemesh-controlplane-hs.easemesh:2380 http://easemesh-control-plane-1.easemesh-controlplane-hs.easemesh:2380]
2021-06-04T11:10:22.206Z INFO cluster/cluster.go:186 starting etcd cluster
2021-06-04T11:10:22.206Z INFO cluster/cluster.go:396 client is ready
2021-06-04T11:10:32.206Z ERROR storage/storage.go:151 pull config failed: context deadline exceeded
2021-06-04T11:10:32.207Z ERROR cluster/cluster.go:237 add self to cluster failed: context canceled
2021-06-04T11:10:32.207Z INFO cluster/config.go:126 etcd config: init-cluster:easemesh-control-plane-0=http://easemesh-control-plane-0.easemesh-controlplane-hs.easemesh:2380,easemesh-control-plane-1=http://easemesh-control-plane-1.easemesh-controlplane-hs.easemesh:2380 cluster-state:existing force-new-cluster:false
2021-06-04T11:10:42.207Z ERROR storage/storage.go:151 pull config failed: context canceled
2021-06-04T11:10:42.211Z INFO storage/storage.go:250 /running_objects.yaml not exist
2021-06-04T11:10:52.214Z ERROR storage/storage.go:151 pull config failed: context canceled
2021-06-04T11:11:02.215Z ERROR storage/storage.go:151 pull config failed: context canceled
2021-06-04T11:11:02.216Z INFO supervisor/supervisor.go:197 create system controller StatusSyncController
2021-06-04T11:11:02.216Z ERROR api/api.go:107 get cluster mutex /config/lock failed: lease is not ready
2021-06-04T11:11:02.217Z ERROR storage/storage.go:313 sync runtime failed: put status failed: lease is not ready
2021-06-04T11:11:02.228Z INFO api/api.go:113 api server running in 0.0.0.0:2381
2021-06-04T11:11:05Z ERROR storage/storage.go:313 sync runtime failed: put status failed: lease is not ready
2021-06-04T11:11:10.001Z ERROR storage/storage.go:313 sync runtime failed: put status failed: lease is not ready
2021-06-04T11:11:15.003Z ERROR storage/storage.go:313 sync runtime failed: put status failed: lease is not ready
2021-06-04T11:11:20Z ERROR storage/storage.go:313 sync runtime failed: put status failed: lease is not ready
Logs in follower:
2021-06-04T11:30:43.546Z INFO cluster/cluster.go:550 hard stop server
2021-06-04T11:30:43.546Z ERROR cluster/cluster.go:255 start server timeout(10m0s)
panic: start server timeout(10m0s)
goroutine 29 [running]:
github.com/megaease/easegress/pkg/cluster.(*cluster).getReady(0xc00032eb00, 0x0, 0x176d4a7)
/home/zhaokun/work/zhao-kun/easegress-1/pkg/cluster/cluster.go:256 +0x5dc
github.com/megaease/easegress/pkg/cluster.(*cluster).run(0xc00032eb00)
/home/zhaokun/work/zhao-kun/easegress-1/pkg/cluster/cluster.go:188 +0xa5
created by github.com/megaease/easegress/pkg/cluster.New
/home/zhaokun/work/zhao-kun/easegress-1/pkg/cluster/cluster.go:158 +0x265
For testing, we need to prepare PV previously. The following PV example was offered to you guys to refer
apiVersion: v1
kind: PersistentVolume
metadata:
name: easemesh-storage-pv-2
spec:
capacity:
storage: 4Gi
# volumeMode field requires BlockVolume Alpha feature gate to be enabled.
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: easemesh-storage
local:
path: /volumes/easemesh-storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- kube-2
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: easemesh-storage-pv-3
spec:
capacity:
storage: 4Gi
# volumeMode field requires BlockVolume Alpha feature gate to be enabled.
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: easemesh-storage
local:
path: /volumes/easemesh-storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- kube-3
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: easemesh-storage-pv-1
spec:
capacity:
storage: 4Gi
# volumeMode field requires BlockVolume Alpha feature gate to be enabled.
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: easemesh-storage
local:
path: /volumes/easemesh-storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- kube-1
The installer/uninstall for EaseMesh
install command
reset command
To help you guys to review the codes, I will explain codes briefly
Modules
I divided the installer into four packages, each package install one of the resources need by the EaseMesh. Each package export four functions which are responsible for:
Deploy
: Deploy deploy one kind of resources need by MeshPreCheck
: PreCheck check prerequisite for needing to be installed resourceClear
: Clear clear all installed resource about the control panelDescribe
: Describe leverage human-readable text to describe a different phase in the process of the resourceAll four exported functions are wrapped as an InstallStage (or filter) object, all stage object was composed as a
Chain of Responsibility
to be invoked one by oneThe package including :
Controller Plane
NodePort
) is dedicated to providing an entrypoints for external component accessing (for example, mesh installer will leverage this port to deployed mesh controller configuration to EG)ClusterIP
) is dedicated to providing mesh sidecar or ingress to communicate with the control planeClusterIP: None
) is dedicated to providing Easegress coordinate with each otherOperator
The operator is a CRD operator, which was deployed as a deployment. The operator is responsible for injecting a sidecar or JavaAgent to a
meshdeployment
resource deployed.Operator resources including:
MeshIngress
MeshIngress is our ingress for mesh, which is an API object that manages external access to the mesh services in a cluster, typically HTTP. MeshIngress resources including:
CRD
CRD is K8s' CustomeResourceDefinition