Independent Vetex/Actor API

[ashione]

Mobius/Ray Streaming might need users create independent actors these out of streaming DAG. In our prevois design, job master will invoke and manage all lifecyle of independent actors, such as Parameter Server actor that supported for training inside.

Actually, different training infra need user-define independent actors so engine dev should design a more general api for creating and managing independent actors/vertices.

ctx = context.build_context(context_config)
ctx.independent_vertex(module_name = 'xx', class_name = 'xx', number = 10, config = {}, resources = {}, lazy=false)

For exmplae, we define a parameter server group:

ctx.independent_vertex(module_name = "penrose.ps_actor", class_name = "PsActor", number  = 10)

[ashione]

@BalaBalaYi we talked offline that api could support linked-declaration likes

ctx.independent_vertex(module_name = "penrose.ps_acotr, class_name = "PsActor")
.with_config({"key" : "value"})
.with_resource({"CPU": 10 , "MEM" : 20})
.set_parallellism(10)
.set_language(PYTHON)
.set_mode(LAZY)

After definition, jobmaster will create 10 parameter servers after dag construction, which also mare use all of workers in DAG can attain ps actor handler then do what by themself.

[BalaBalaYi]

Still working on it. Will sync the impl to the repo after the internal release.