Support native deployment

[xxx7xxxx]

Detail

Actually, this is also a refactoring in implementing the new feature:

1. Simplify interface with a lot of arguments.
2. Complete service spec.
3. Simplify complex containers/volumes layout (see reference comparison), moreover, it is more consistent.
4. Make logs clean and consistent.

And there left something to complete, which of part were commented, and:

1. Add documents for supporting native deployment.
2. Add more unit tests.

After them, I found more trivial or significant stuff needed to be improved, which will be opened issues in the next contribution.

Reference

There lists the old and new specs of deployment after injecting the sidecar. The same/unimportant part has been committed. We can see the two init containers have been merged into one, and the volumes and volumeMounts became more consistent. The config of k8s objects are complex enough, it's very important to manage the complexity of them carefully.

old-vets-service.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vets-service
  namespace: spring-petclinic
spec:
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: vets-service
    spec:
      containers:
      - args:
        - -c
        - java -server -Xmx1024m -Xms1024m -Dspring.profiles.active=sit -Djava.security.egd=file:/dev/./urandom  org.springframework.boot.loader.JarLauncher
        command:
        - /bin/sh
        env:
        - name: JAVA_TOOL_OPTIONS
          value: ' -javaagent:/easeagent-volume/easeagent.jar -Deaseagent.log.conf=/easeagent-volume/log4j2.xml '
        image: megaease/spring-petclinic-vets-service:latest
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - sleep 10
        name: vets-service
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: "2"
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 256Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /application/application-sit.yml
          name: configmap-volume-0
          subPath: application-sit.yml
        - mountPath: /easeagent-volume
          name: easeagent-volume
      - command:
        - /bin/sh
        - -c
        - /opt/easegress/bin/easegress-server -f /easegress-sidecar/eg-sidecar.yaml
        env:
        - name: APPLICATION_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        image: megaease/easegress:server-sidecar
        imagePullPolicy: Always
        name: easemesh-sidecar
        ports:
        - containerPort: 13001
          name: sidecar-ingress
          protocol: TCP
        - containerPort: 13002
          name: sidecar-egress
          protocol: TCP
        - containerPort: 13009
          name: sidecar-eureka
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /easegress-sidecar
          name: sidecar-params-volume
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /bin/sh
        - -c
        - cp -r /easeagent-volume/. /easeagent-share-volume
        image: megaease/easeagent-initializer:latest
        imagePullPolicy: Always
        name: easeagent-initializer
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /easeagent-share-volume
          name: easeagent-volume
      - command:
        - /bin/sh
        - -c
        - |-
          echo name: $POD_NAME >> /opt/eg-sidecar.yaml; echo 'cluster-join-urls: http://easemesh-controlplane-svc.easemesh:2380
          cluster-request-timeout: 10s
          cluster-role: reader
          cluster-name: easemesh-control-plane
          Labels:
            alive-probe: http://localhost:9900/health
            application-port: "8080"
            mesh-service-labels: ""
            mesh-servicename: vets-service
          ' >> /opt/eg-sidecar.yaml; cp -r /opt/. /sidecar-params-volume
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: megaease/easegress:server-sidecar
        imagePullPolicy: Always
        name: easegress-sidecar-initializer
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sidecar-params-volume
          name: sidecar-params-volume
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: application-sit-yml
            path: application-sit.yml
          name: vets-service
        name: configmap-volume-0
      - emptyDir: {}
        name: easeagent-volume
      - emptyDir: {}
        name: sidecar-params-volume

new-vets-service.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vets-service
  namespace: spring-petclinic
spec:
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: vets-service
    spec:
      containers:
      - args:
        - -c
        - java -server -Xmx1024m -Xms1024m -Dspring.profiles.active=sit -Djava.security.egd=file:/dev/./urandom  org.springframework.boot.loader.JarLauncher
        command:
        - /bin/sh
        env:
        - name: JAVA_TOOL_OPTIONS
          value: ' -javaagent:/agent-volume/easeagent.jar -Deaseagent.log.conf=/agent-volume/log4j2.xml '
        image: megaease/spring-petclinic-vets-service:latest
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - sleep 10
        name: vets-service
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: "2"
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 256Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /application/application-sit.yml
          name: configmap-volume-0
          subPath: application-sit.yml
        - mountPath: /agent-volume
          name: agent-volume
      - command:
        - /bin/sh
        - -c
        - /opt/easegress/bin/easegress-server -f /sidecar-volume/sidecar-config.yaml
        env:
        - name: APPLICATION_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        image: megaease/easegress:server-sidecar
        imagePullPolicy: IfNotPresent
        name: easemesh-sidecar
        ports:
        - containerPort: 13001
          name: sidecar-ingress
          protocol: TCP
        - containerPort: 13002
          name: sidecar-egress
          protocol: TCP
        - containerPort: 13009
          name: sidecar-eureka
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sidecar-volume
          name: sidecar-volume
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - |-
          set -e
          cp -r /easeagent-volume/* /agent-volume

          echo 'name: vets-service
          cluster-join-urls: http://easemesh-controlplane-svc.easemesh:2380
          cluster-request-timeout: 10s
          cluster-role: reader
          cluster-name: easemesh-control-plane
          labels:
            alive-probe:
            application-port: 8080
            mesh-service-labels:
            mesh-servicename: vets-service
          ' > /sidecar-volume/sidecar-config.yaml
        image: megaease/easeagent-initializer:latest
        imagePullPolicy: Always
        name: initializer
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /agent-volume
          name: agent-volume
        - mountPath: /sidecar-volume
          name: sidecar-volume
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: application-sit-yml
            path: application-sit.yml
          name: vets-service
        name: configmap-volume-0
      - emptyDir: {}
        name: agent-volume
      - emptyDir: {}
        name: sidecar-volume

The diff of them:

diff --git old-vets-service.yaml new-vets-service.yaml
index cf4331e..dcb0781 100644
--- old-vets-service.yaml
+++ new-vets-service.yaml
@@ -18,7 +18,7 @@ spec:
         - /bin/sh
         env:
         - name: JAVA_TOOL_OPTIONS
-          value: ' -javaagent:/easeagent-volume/easeagent.jar -Deaseagent.log.conf=/easeagent-volume/log4j2.xml '
+          value: ' -javaagent:/agent-volume/easeagent.jar -Deaseagent.log.conf=/agent-volume/log4j2.xml '
         image: megaease/spring-petclinic-vets-service:latest
         imagePullPolicy: IfNotPresent
         lifecycle:
@@ -45,12 +45,12 @@ spec:
         - mountPath: /application/application-sit.yml
           name: configmap-volume-0
           subPath: application-sit.yml
-        - mountPath: /easeagent-volume
-          name: easeagent-volume
+        - mountPath: /agent-volume
+          name: agent-volume
       - command:
         - /bin/sh
         - -c
-        - /opt/easegress/bin/easegress-server -f /easegress-sidecar/eg-sidecar.yaml
+        - /opt/easegress/bin/easegress-server -f /sidecar-volume/sidecar-config.yaml
         env:
         - name: APPLICATION_IP
           valueFrom:
@@ -58,7 +58,7 @@ spec:
               apiVersion: v1
               fieldPath: status.podIP
         image: megaease/easegress:server-sidecar
-        imagePullPolicy: Always
+        imagePullPolicy: IfNotPresent
         name: easemesh-sidecar
         ports:
         - containerPort: 13001
@@ -74,52 +74,39 @@ spec:
         terminationMessagePath: /dev/termination-log
         terminationMessagePolicy: File
         volumeMounts:
-        - mountPath: /easegress-sidecar
-          name: sidecar-params-volume
+        - mountPath: /sidecar-volume
+          name: sidecar-volume
       dnsPolicy: ClusterFirst
       initContainers:
       - command:
-        - /bin/sh
-        - -c
-        - cp -r /easeagent-volume/. /easeagent-share-volume
-        image: megaease/easeagent-initializer:latest
-        imagePullPolicy: Always
-        name: easeagent-initializer
-        resources: {}
-        terminationMessagePath: /dev/termination-log
-        terminationMessagePolicy: File
-        volumeMounts:
-        - mountPath: /easeagent-share-volume
-          name: easeagent-volume
-      - command:
-        - /bin/sh
+        - sh
         - -c
         - |-
-          echo name: $POD_NAME >> /opt/eg-sidecar.yaml; echo 'cluster-join-urls: http://easemesh-controlplane-svc.easemesh:2380
+          set -e
+          cp -r /easeagent-volume/* /agent-volume
+
+          echo 'name: vets-service
+          cluster-join-urls: http://easemesh-controlplane-svc.easemesh:2380
           cluster-request-timeout: 10s
           cluster-role: reader
           cluster-name: easemesh-control-plane
-          Labels:
-            alive-probe: http://localhost:9900/health
-            application-port: "8080"
-            mesh-service-labels: ""
+          labels:
+            alive-probe:
+            application-port: 8080
+            mesh-service-labels:
             mesh-servicename: vets-service
-          ' >> /opt/eg-sidecar.yaml; cp -r /opt/. /sidecar-params-volume
-        env:
-        - name: POD_NAME
-          valueFrom:
-            fieldRef:
-              apiVersion: v1
-              fieldPath: metadata.name
-        image: megaease/easegress:server-sidecar
+          ' > /sidecar-volume/sidecar-config.yaml
+        image: megaease/easeagent-initializer:latest
         imagePullPolicy: Always
-        name: easegress-sidecar-initializer
+        name: initializer
         resources: {}
         terminationMessagePath: /dev/termination-log
         terminationMessagePolicy: File
         volumeMounts:
-        - mountPath: /sidecar-params-volume
-          name: sidecar-params-volume
+        - mountPath: /agent-volume
+          name: agent-volume
+        - mountPath: /sidecar-volume
+          name: sidecar-volume
       restartPolicy: Always
       schedulerName: default-scheduler
       securityContext: {}
@@ -133,6 +120,6 @@ spec:
           name: vets-service
         name: configmap-volume-0
       - emptyDir: {}
-        name: easeagent-volume
+        name: agent-volume
       - emptyDir: {}
-        name: sidecar-params-volume
+        name: sidecar-volume

[zhao-kun]

It's good to refactor about reducing numbers of initializing containers

[xxx7xxxx]

Updated as comments along with ImagePullPolicy assigned with IfNotPresent by default

[xxx7xxxx]

scenarios 0

You seem not to delete the meshdeployment with the same name.

scenarios 1

According to the quick research, it seems we don't have that interface to block the system-level controller to deploy Deployment immediately after it was creating. It seems not to give us the ability to register pre-hook.

scenarios 2

kubectl replace will work fine instead of kubectl apply, the mechanism of two interfaces decides the different results[1]. We add more fields for the original spec, so applying part of spec will cause the Not Found issue.

Could we organize the canary labels in the mesh annotations?

Sure, the example could be:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: spring-petclinic
  name: vets-service
  annotations:
    mesh.megaease.com/enable: "true"
    mesh.megaease.com/service-name: "vets-service"
    mesh.megaease.com/service-labels: "version=beta,env=production"

It seems harder than structural labels to write and read, but acceptable for me.

[1] https://stackoverflow.com/a/62067419/1705845

[zhao-kun]

FYI: Other mesh products leverage Dynamic Admission Control to modify objects sent to the API server to enforce custom defaults.

[xxx7xxxx]

Updated with Dynamic Admission Control.

[zhao-kun]

There are some conflicts should be resolved

[xxx7xxxx]

Conflict fixed. I think it's cool to support more objects but this PR is big enough, I'd like to support them in my next contribution.