search

CentaurusInfra / fornax

Fornax for autonomous and flexible edge computing
Apache License 2.0
8 stars 16 forks source link

resync missions in edgeclusters periodically #11

Closed chenqianfzh closed 3 years ago

chenqianfzh commented 3 years ago

With this PR, the edgecontroller will align the missions in edgeclusters with the missions in the cloud side.

In more details: Every time an edgecluster object is added or updated, edgecontroller will look into the missions deployed in the edge cluster. If there is any discrepancy between the missions of the edge side and the cloud side, the edge controller will send an MissionList message to the ege side. The MetaManager will update the mission records saved in its DB after receiving the MissionList message. Finally, the clusterd will pick up the changes and update the missions in its managed edge cluster.

Verification

The PR is verified with the following test scenario :

  1. Start the cloudcore

  2. keep the edgecore off and deploy a new mission A

  3. start edgecore It is verified that the mission A is deployed to the edge cluster

  4. stop the edgecore process

  5. delete the mission A and deploy a new mission B in the cloud cluster.

  6. restart the edgecore process, wait like 20 seconds

  7. It is verified that in the edge cluster, mission A is removed and mission B is active.

  8. Manually delete the mission B in the edge cluster.

  9. wait 20 seconds.

  10. It is verified that mission B is re-deployed and still active in the edge cluster.

Possible improvement

In this PR, the mission list message will be sent out only when it is necessary, namely when the edge controller finds the missions in the edge is different from the missions in the cloud side. For the PoC purpose, I compare the missions of edge and cloud by their names. Therefore, if a mission's spec is manually changed in the edge side with the mission name untouched, the cloud side cannot detect it.

I am thinking this issue might be solved by clusterd sending a Mission Spec hash value in the edge-cluster status message, where the edgecontroller get the information of missons running on the edge cluster. Then the edge controller can also check whether the mission Spec is different from that in the cloud side. It is left for further discussion .

pdgetrf commented 3 years ago

@jshaofuturewei this is POC code so please ignore the test failures and just focus on ideas being verified.