Closed haoel closed 2 years ago
The shadow service needs to support the creation of requests from other services, such as easeload, or users through command-line operations. Therefore, it needs to:
In addition to the deployment object created by meshdeployment, easemesh also supports native deployment, so we only need to copy the deployment. Copy all or a single qualified deployment of a namespace by specifying the namespace and deployname.
Deployment replication rules are as follows:
Examples are as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"mesh.megaease.com/alive-probe-url":"http://localhost:9900/health","mesh.megaease.com/service-name":"vets-service"},"name":"vets-service","namespace":"spring-petclinic"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"vets-service"}},"template":{"metadata":{"labels":{"app":"vets-service"}},"spec":{"containers":[{"args":["-c","java -server -Xmx1024m -Xms1024m -Dspring.profiles.active=sit -Djava.security.egd=file:/dev/./urandom org.springframework.boot.loader.JarLauncher"],"command":["/bin/sh"],"image":"megaease/spring-petclinic-vets-service:latest","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sh","-c","sleep 10"]}}},"name":"vets-service","ports":[{"containerPort":8080}],"resources":{"limits":{"cpu":"2000m","memory":"1Gi"},"requests":{"cpu":"200m","memory":"256Mi"}},"volumeMounts":[{"mountPath":"/application/application-sit.yml","name":"configmap-volume-0","subPath":"application-sit.yml"}]}],"restartPolicy":"Always","volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"application-sit-yml","path":"application-sit.yml"}],"name":"vets-service"},"name":"configmap-volume-0"}]}}}}
mesh.megaease.com/alive-probe-url: http://localhost:9900/health
mesh.megaease.com/service-name: vets-service
name: vets-service
namespace: spring-petclinic
spec:
replicas: 1
selector:
matchLabels:
app: vets-service
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: vets-service
spec:
containers:
- args:
- -c
- java -server -Xmx1024m -Xms1024m -Dspring.profiles.active=sit -Djava.security.egd=file:/dev/./urandom org.springframework.boot.loader.JarLauncher
command:
- /bin/sh
env:
- name: JAVA_TOOL_OPTIONS
value: ' -javaagent:/agent-volume/easeagent.jar -Deaseagent.log.conf=/agent-volume/log4j2.xml '
image: megaease/spring-petclinic-vets-service:latest
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- sh
- -c
- sleep 10
name: vets-service
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 200m
memory: 256Mi
volumeMounts:
- mountPath: /application/application-sit.yml
name: configmap-volume-0
subPath: application-sit.yml
- mountPath: /agent-volume
name: agent-volume
- command:
- /bin/sh
- -c
- /opt/easegress/bin/easegress-server -f /sidecar-volume/sidecar-config.yaml
env:
- name: APPLICATION_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
image: 172.20.2.189:5001/megaease/easegress:server-sidecar
imagePullPolicy: IfNotPresent
name: easemesh-sidecar
ports:
- containerPort: 13001
name: sidecar-ingress
protocol: TCP
- containerPort: 13002
name: sidecar-egress
protocol: TCP
- containerPort: 13009
name: sidecar-eureka
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /sidecar-volume
name: sidecar-volume
initContainers:
- command:
- sh
- -c
- "set -e\ncp -r /easeagent-volume/* /agent-volume\n\necho 'name: vets-service\ncluster-join-urls:
http://easemesh-controlplane-svc.easemesh:2380\ncluster-request-timeout:
10s\ncluster-role: reader\ncluster-name: easemesh-control-plane\nlabels:\n
\ alive-probe: http://localhost:9900/health\n application-port: 8080\n
\ mesh-service-labels: \n mesh-servicename: vets-service\n' > /sidecar-volume/sidecar-config.yaml"
image: 172.20.2.189:5001/megaease/easeagent-initializer:latest
imagePullPolicy: IfNotPresent
name: initializer
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /agent-volume
name: agent-volume
- mountPath: /sidecar-volume
name: sidecar-volume
volumes:
- configMap:
defaultMode: 420
items:
- key: application-sit-yml
path: application-sit.yml
name: vets-service
name: configmap-volume-0
- emptyDir: {}
name: agent-volume
- emptyDir: {}
name: sidecar-volume
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"mesh.megaease.com/alive-probe-url":"http://localhost:9900/health","mesh.megaease.com/service-name":"vets-service"},"name":"vets-service","namespace":"spring-petclinic"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"vets-service"}},"template":{"metadata":{"labels":{"app":"vets-service"}},"spec":{"containers":[{"args":["-c","java -server -Xmx1024m -Xms1024m -Dspring.profiles.active=sit -Djava.security.egd=file:/dev/./urandom org.springframework.boot.loader.JarLauncher"],"command":["/bin/sh"],"image":"megaease/spring-petclinic-vets-service:latest","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sh","-c","sleep 10"]}}},"name":"vets-service","ports":[{"containerPort":8080}],"resources":{"limits":{"cpu":"2000m","memory":"1Gi"},"requests":{"cpu":"200m","memory":"256Mi"}},"volumeMounts":[{"mountPath":"/application/application-sit.yml","name":"configmap-volume-0","subPath":"application-sit.yml"}]}],"restartPolicy":"Always","volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"application-sit-yml","path":"application-sit.yml"}],"name":"vets-service"},"name":"configmap-volume-0"}]}}}}
mesh.megaease.com/alive-probe-url: http://localhost:9900/health
mesh.megaease.com/service-name: vets-service
// 1. Add shadow service-label
mesh.megaease.com/service-labels: "version=shadow"
// 2. change Deployment name
name: vets-service-shadow
namespace: spring-petclinic
spec:
replicas: 1
selector:
matchLabels:
app: vets-service
// 3. add new selector for shadow pods
mesh.megaease.com/mesh-shadow-service: true
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: vets-service
// 4. add new label for shadow pods
mesh.megaease.com/mesh-shadow-service: true
spec:
containers:
- args:
- -c
- java -server -Xmx1024m -Xms1024m -Dspring.profiles.active=sit -Djava.security.egd=file:/dev/./urandom org.springframework.boot.loader.JarLauncher
command:
- /bin/sh
env:
- name: JAVA_TOOL_OPTIONS
value: ' -javaagent:/agent-volume/easeagent.jar -Deaseagent.log.conf=/agent-volume/log4j2.xml '
image: megaease/spring-petclinic-vets-service:latest
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- sh
- -c
- sleep 10
name: vets-service
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 200m
memory: 256Mi
volumeMounts:
- mountPath: /application/application-sit.yml
name: configmap-volume-0
subPath: application-sit.yml
volumes:
- configMap:
defaultMode: 420
items:
- key: application-sit-yml
path: application-sit.yml
name: vets-service
name: configmap-volume-0
@haoel @zhao-kun @xxx7xxxx
Let me clarify the shadow service architecture through a diagram:
Shadow service: It is an in-cluster service, which accepts requests forward from the EaseMesh control plane (Easegress). When a request is incoming, the Shadow service will fetch the deployment information in the K8s cluster, After rendering a new shadow deployment with annotation, it will send create to the K8s API server to deploy new deployments.
MeshController: It's the endpoint of the EaseMesh control plane, which manages the specifications and service registry of the Mesh. The emctl
and other services could integrate EaseMesh via it.
First, The logic implemented by the Shadow service doesn't belong to the scope of the common Mesh. Second, it's heavily communicated with K8s and very different from EaseMesh controller implementation. Thirdly, like K8s, the EaseMesh control plane needs a mechanism to allow developers to customize the EaseMesh. After several days of study, I found the K8s has a mechanism to extend the K8s API server which is API Aggregation layer [1]. I think we could leverage its design extending the EaseMesh control plane and implement the requirement.
[1]https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation
Some Q&A:
We use k8s apiserver as one and only one gate for all APIs, and MeshController will also support extensional APIs, so it's better to use unique access gate.
No, the extensible APIs of MeshController will cover it from start.
It depends, we need to design a general APIs(Mesh service dedicated info, general storage, etc) for customized serviced.
Related to #107, It will be closed after #109 is accepted
This is a requirement is needed by performance testing on production.
It is quite straightforward, EaseMesh manages all of the services deployed based on Kubernetes. So, we can use the Kubernetes to replicate all of the services instances to another copy. We call this the "Shadow Service". After that, we can schedule the test traffic to the "shadow service" for the test.
In other words, we try to finish the following works.
Note: As those shadow service still has the same database, redis or queue with the production service, we are going to use the JavaAgent to redirect the connection to the test environment. This requirement is addressed by https://github.com/megaease/easeagent/issues/99