[PROPOSAL] More programmer friendly beehive framework

[tedli]

What would you like to be added:

Please refer to this demo: https://github.com/tenxcloud/serviced/blob/master/example/main.go

type EdgeCore interface {
    CreatePod(pod Pod) (result PodCreationResult)
    // ...... other methods
}

type EdgeD interface {
    CreatePod(pod Pod) (status PodStatus)
    // ...... other methods
}

type edgeCoreImpl struct {
    EdgeDModule EdgeD  // can be injected
    // ...... other dependencies
}

func (eci edgeCoreImpl) CreatePod(pod Pod) (result PodCreationResult) {
    status := eci.EdgeDModule.CreatePod(pod)
    result := status.ToResult()
    return
}

// at edgecontroller side

edgeCore := kernel.ResolveService(new(EdgeCore))

// resolves out a transparent proxy which convert method call to message then send
// through websocket or other transport object.
result := edgeCore.CreatePod(theNewCreatePodFromEvent)

Why is this needed:

beehive framework now has points that now good enough:

1. Hard for one who new to kubeedge code to understand interactions between modules (without accurate sequence diagram);
2. Hard to debug (nor do unit test);
3. Easy to make a mistake;
4. A module can 'Receive' messages which should not be received by this module.

[sids-b]

cc @qizha @m1093782566

[qizha]

Actually Beehive is designed as a in-process messaging system and pluggable module support system instead of a RPC system. The module is new concept in beehive just like microservices on cloud. The unnecessary modules can be closed with configuration to reduce the resource cost. Beehive is designed to handle the cross-module communication. At the beginning we only have go-channel communication cross modules. Later, we have few requirements to provide corss-process module communication so we add socket/UDS capability but that part is not in kubeedge. So we only need in-process communication with go-channel in kubeedge now. The idea is to use messaging system to make the modules running in one process be pluggable. The message can be send to one module or multiple modules. We group each modules during initialization so every modules in this group can get messages if one module send message to this group. This is useful to de-couple modules. For example, we have one group named "hub" and there is one module "edgehub" right now. Now we are planning to add quic support by adding a new module someting like "edgehub-quic". What we need to do is just add this new module and add it to "hub" group, that's all. No one else need to be aware of this new guy. The samething happen on cloud part. But the communication between cloud and edge is rely on cludhub/edgehub instead of the Beehive. But yes I agree with you we need a document to describe the interaction between modules, that will make things clearly. BTW, you may want to check the real microservice framework go-chassis(https://github.com/go-chassis/go-chassis) which comes from my previous project.

[tedli]

@qizha Thanks for reply. The proposal not intend to re-implement a rpc system. It just learnt from beehive to give a pluggable module support and inter process message call system.

The module in beehive could be considered as a Service, which expose a set of methods. By sending message to a module or a group, a method (a set of methods if send message to a group) can be invoked according to routing policy.

So 'module' could be contracted as an interface. By giving an implementation and register the implementation to the the Service, any other module which depends on this module (want to send message to this module) can resolve out an object that implemented the Service, and then call the method (send message to it). If the implementation is on server side, the object is a proxy (only in this case it do rpc things).

Also the proposal provide a dependencies injection system. When a module depends on other modules, the dependencies can be injected.

As the main propose of beehive is pluggable, what if no 'eventbus' or 'edged' in edge core's config?

modules:
    enabled: [eventbus, websocket, metaManager, edged, twin, dbTest]

The program may not run properly. Then if used some system like this proposal want to implement, the dependency (event bus) is a proxy, other process which implements eventbus can be called.

The demo is for proposal not a production level package.

Beehive is great, and inspired by beehive, we think it could do more to clarify interactions between moduls by code itself not document which is always late come out and outdated.

[sids-b]

@tedli , If a module(say X) has to contacted by an interface.... other modules wanting to contact X need to be aware of the interface implemented by X right ? If yes, assuming we will have 10 modules in future, each of them need to aware about 9 other interfaces implemented by other modules. Also adding a new module would need source code changes for making existing modules aware about the new modules interface.

Please correct me if I am wrong

[m1093782566]

xref: http://www.sel.zju.edu.cn/?p=989

[tedli]

@sids-b Let us forget about this proposal and current implementation of beehive for now. If module A want's to send message to module B, A have to know there is a module B, just that A not need to know how B implemented. If A even don't know there is a module B why A wants to send message to.

So answer your question, no matter there are 10 modules or more, if a module depends on other modules (it wants send messages to ), it has to know the modules it depends on, just not need to care about the implementation.

And for now on, modules on controller side or edge side now have to co-exists. If no 'eventbus' at edge core, it just doesn't work. It's not about pluggable, it's dependencies.

The proposal want's to inject interfaces (modules [it's dependencies]) to a struct (a implementation of other interface [a module]), while it not need to care about the interfaces' implementations, the implementation may be a proxy if the implementation not in the same process.

[tedli]

The 'in-process communication' indeed acts in asynchronous manner. But async also can be done in many other ways.

For current beehive implementation, in process communication just act a complex method call. I'm not saying it's not good. It‘s just about preference.

[sids-b]

@tedli , as @qizha said, module A doesn't want to send message to module B. It wants to send it to a group. Each group has a responsibility. For example we have one group named "hub" and there is one module "edgehub" right now. Now we are planning to add quic support by adding a new module someting like "edgehub-quic". What we need to do is just add this new module and add it to "hub" group, that's all. No one else need to be aware of this new guy. So module A sends message to Hub group and not module B(which can be edgeHub). Module A does not care who is present is Hub group..... The responsibility of Hub group is to send message to cloud.

[tedli]

@sids-b Indeed. Module A depends on Module B's contract (an interface). Let's named this interface 'hub'. 'edgehub-quic' is nothing but an implementation of Module B's contract. Module A also not need to know this new guy, it just a 'hub' interface (but now is 'edgebub-quic' version). Group of beehive is brilliant. That's why I was inspired so much. And all these are also configurable by 'module.yaml'.

[sids-b]

@tedli Suppose the current implementation of Hub is

type Hub interface {
    A()
    B()
}

In the future version I change it to

type Hub interface{
    C()
    D()
}

With current implementation, there is no need for module A to be aware of change in interface and change in function definition as well :). Indeed an amazing design by @qizha :)

[tedli]

@sids-b For current beehive implementation, what if a module's routing policies changed.

That's the same thing about method changes, which the former will cause errors or panic in runtime, the latter can be checked by compiler during compile time.

[sids-b]

I don't think there will be runtime errors and panics as long as Hub Group design owner make sure C() is backward compatible with A() and D() is backward compatible with B(). Backward compatibility is always expected in an interface upgrade :)

[sids-b]

Requesting @m1093782566 and @qizha to put their thoughts.

[tedli]

'Go module' can import more than one version of type.

// github.com/kubeedge/beehive/hub/v1
type Hub interface {
    A()
    B()
}

// github.com/kubeedge/beehive/hub/v2  <---
type Hub interface {
    A()
    B()
    C()
    D()
}

A v2 version of Hub implementation is also a v1 one.

Just a proposal for ones who loves beehive and kubeedge and want to do further contributions like me to contribute without any pre-release docs or a private internal company meeting. Not mean to replace or diss the current implementation.

Anyway it's all about preference.

[nkwangleiGIT]

@tedli @m1093782566 @qizha It's just some initial thought about current in-process messaging design of beehive, and I think we can talk about this later next week, it should be more efficient. For now, we should pay more efforts to fix bugs and enable more functionalities of young KubeEdge :-)

Thanks all for the discussion above, and will update here after our offline communication.

[sids-b]

@tedli , Please don't get me wrong that I am opposing this proposal. I am only trying to understand it better. Glad to know that you like KubeEdge and beehive :) Looking forward to more collaborative work to improve both KubeEdge and beehive :)

[qizha]

@tedli @nkwangleiGIT I am very glad to talk with you guys about the design ideas. And I'd like to share with you the reasons why beehive implemented like this. Also, I'd like to see we can make it better together ^_^ This is a big topic actually. I know exactly what @tedli is proposing because I built ServiceComb(https://servicecomb.apache.org/cn/) in the last two years as well as the "microservice infrastructure as a service" in production. What @tedli is trying to implement here is something like dynamic proxy in Java and ServiceComb\Dubbo use this mechanism to implement their microservice calling system. I am very impressed you implement it with the language not that dynamically like golang. It looks great! Just like tedli mentioned, let's back to the requirement itself. Here are some features we need for this system:

1. We need not only 1:1 communication but 1:n. For example, a connection lost event shoud be propagate to multiple consumers on edge to handle such situation. This is more like a MQ style communication system then.
2. Everything should happen asynchronously with a reactive programming style to accelerate the operation.
3. The changes happened to each module should not impact the others. The changes here means code changes or add/remove modules even in runtime. So we need a contract-oriented system instead of interface-oriented. The message wrapper is that contract then: https://github.com/kubeedge/beehive/blob/master/pkg/core/model/message.go. However, the dynamic plugin in golang is not mature enough so we have to include every modules when doing compiling(we can consider to use golang dynamic plugging to implement the module online adding/removing feature without impacting the running system). But user can close/open some of them even in runtime by balancing the functionalities and resource cosuming. That include edged(user can shutdown container deployment and keep the function implement by another module) and eventbus(user can use servicebus to communicate) actually. Sorry for the missleading that we are not open source function and servicebus in the moment because we want to implement function with Knative in the community and servicebus is coming soon.
4. Easy for testing and mockup. Remember the contract-oriented here, the contract is the message wrapper regardless the protocol so we can use go-channel on Edge and websocket/quic between edge and cloud. Each developer work on different modules can use the same mechanism to communicate with others then. And it is easy for them to create mock modules to test their module by mockup the contract.
5. Later we found we need the same communication mechanism cross processes and languages. See we may have some monitoring agent implemented by another language running separately. And each modules in kubeedge may need to send information to it.

Well, that's what we need. And then I realized what I built in the last two years may not be suitable to handle such situation. A MQ style messaging system plus a well wrapped message format maybe better. But I do agree with you that we should find a way to make such message interactivtion be declarable instead of in the code. See we may have one file for each module to declare the message interaction relationship so that everyone can easily get the whole system module interaction relationship from these files. What's your take?