Some confusing about Instruction Implementation

[qazqwe1596357]

Some confusing about Instruction Implementation

My knowledge of Instruction

• gleam Now Instruction Implementation In this ways
  • 1 - Implements one Class that implements the Instruction interface. The Instruction Class may has some configuration context variables.
  • 2 - Implements one pb proto description in pb/gleam.proto Instruction substruct. The description can hold all configuration context for this Instruction.
  • 3 - Register the Instruction by Register one lambda func that can generate this Instruction if the pb-proto object match this Instruction.
• gleam use Instruction this ways
  • 1 - Step set the Instruction with some configuration parameters, some may from APP programs.
  • 2 - Instruction can serialize this parameters into pb bytes, and can rebuild the Instruction in another task process.
  • 3 - The way rebuild the Instruction, by try each "pb to Instruction lambda func registered", if build success, that is it.

My confusing about Instruction Implementation

• This realization is a bit tortuous, There is too much coupling.

  • A new Instruction must MODIFY the hard encoded pb proto file, and regenerate the pb go source.
  • Most Implementations of Instructions, the Instruction desc in pb proto file, indeed is the Instruction configuration context
  • But we write twice in Instruction struct and pb file, and must keep they are compatible
  • Which Instruction current is rebuilding, is by try rebuild function one by one, try who can run success

• In this way, Add Instruction is some "Arduous Engineering"

• User cannot ADD new Instruction with some User expected router shards communication in Use Application Space, Or without modify the pb file.

• I think the Instruction Implementation actually can be implemented in the same way like the Mapper / Reducer

  • The Instruction can be register by Instruction ID
  • And the Instruction configuration context can serialization and deserialization in JSON like or what the Instruction like to. Because just the Instruction itself need to konw it.
  • In this way, One Instruction Implementation just want to implement the function it need, and serialization and deserialization method, can register anywhere.

• So My confusing about Instruction Implementation is why use current Implementation, Are there any special considerations?

[chrislusf]

Thanks! Your understanding is accurate. Originally there are only Instructions. The Mapper/Reducer design came later. It should be possible to simplify the instructions similar to mapper/reducer.

[qazqwe1596357]

Thanks I Got it. I keep learning gleam architecture, The vast majority of the design of gleam is particularly good. I hope to be able to push more code in the future~

[qazqwe1596357]

current realizing of Instruction

• I saw the pb.Instruction is something that "Gleam Executor" only can run in current design.
• And indeed, the Executor binary file current is the "Gleam Agent" binary file.
• So Some dynamically Functions defined in Driver is not posiable pass to Executor to run.
• And the mechanism Mapper/Reducer(same shell or maybe Lua scripts) realization, something like a patch to support the Go Functions, which add some judge logics(judge have go, is pipes, and so on) in many where, and copy the Driver as one resource file, and then run it use one command descript in Instruction, and use the Mapper/Reducer Register. And this only can use in one InputChan to One OutputChan cases.

ideas for Executor

• So in my opinion, these maybe caused by "The Executor has no context of the Driver", so we design many ways to pass it.
• How about the idea of Just let the Driver Be the Executor?
  • When run Task remote, the agent build the task environment
  • then copy the driver
  • then run the driver with the "Task for current Executor" args
  • we add hook at the Flow.Run(), judge the args whether current run is Driver or a Task, and pick up the Task Instruction context, run the Step.Function()
  • because Datasets/Steps have inited, so the DriverExecutor have all same Steps as Driver, just pick the right Funcion in right Step
  • in this way, no matter what function it is, is more flexible to realize user functions