Extend mission

[chenqianfzh]

This PR, combined with PR https://github.com/CentaurusInfra/fornax/pull/40, provides a solution to deploy the components of an AI face recognition app to fornax edge-clusters, both Arktos and Vanilla Kubernetes.

This PR is about the code logic changes, while PR 40 gives the mission yaml files used to deploy the app .

Motivation

The manual operation of this AI app is a bit cumbersome. The detailed procedures to do it in Arktos and K8s clusters are given in https://github.com/chenqianfzh/kube-cluster-sample/wiki/Procedure-to-Set-Up-A-Distributed-Face-Recognition-Application-in-Arktos and https://github.com/chenqianfzh/kube-cluster-sample/wiki/Procedure-to-Set-Up-A-Distributed-Face-Recognition-Application-in-Vanilla-Kubernetes.

This work make the app deployment a breeze. The simple & easy process is given in the section of "Verification".

Explanation about the Code change

The main goal of this PR is to allow a mission to specify a command to run in edge clusters. Previously, the content of mission has to be the yaml content of a K8s/arktos resource.

I will send out another PR to update the doc to explain the design involved in this PR. As for ow, to make code reading a bit easier, I split the code in this PR into multiple commits per code logic:

1. the code change to the struct of MIssion
2. code change in edgecore to run mission command
3. run a background mission command
4. support the the value of edgecluser kubedistro as a built-in label, which make the tailored mission content per KubeDistro easier.
5. fixing the bugs I hit during the work

Verification Done

The following commands, if not specified, are run under the root directory of fornax repo.

Test Environment Preparation

1. prepare three AWS machine, A, B, C.

2. deploy kubeadm in the machine A.

3. start cloudcore in machine A.

4. start a V1.19 Kubernetes test cluster in machine B, with command

   ENABLE_HOSTPATH_PROVISIONER=true hack/local-up-cluster.sh

5. start arktos test cluster in machine B

   ENABLE_HOSTPATH_PROVISIONER=true hack/arktos-up.sh

6. Configure the /etc/kubeedge/etc/edgecore.yaml config files n machine B t and Co connect to machine A as a K8s and Arktos edge cluster respectively. Details are available in doc https://github.com/CentaurusInfra/fornax/blob/main/docs/fornax_test/test_clusters_setup.md

Tests Done

demo video: https://github.com/pdgetrf/ArktosEdge/blob/main/slides/sept_16_team_demo.mp4

1. deploy the yaml files presented in PR 40

   
   kubectl apply -f tests/edgecluster/data/ai_app/

2.  Let the edge clusters in machine B and C connecto to cloudcore in machine A by running 

_output/local/bin/edgecore --edgecluster

3. Wait 2-3 minutes. The edge clusters need some time to pull down the missions and deploy the contents.
4. Verify the app is working. In machine B and C:
Run command

curl -d '{"path":"/unknown_people/unknown.jpg"}' http://127.0.0.1:8000/image/post

You should get a successful return like: 

got path: {Path:/unknown_people/unknown.jpg}
got paths: {Paths:[{Path:/unknown_people/unknown.jpg}]}
image saved with id: 1 image sent to nsq

Start a client console to mysql using command

mysql -ptcp -h127.0.0.1 -P3306 -uroot -ppassword123

You should get a similar output when you run the sql command as below, which shows this request is processed:

mysql> use kube;

mysql> select i.path, p.name from images i join person p where i.person = p.id; +-----------------------------+----------+ | path | name | +-----------------------------+----------+ | /unknown_people/unknown.jpg | Hannibal | +-----------------------------+----------+ 1 row in set (0.00 sec)