fission / fission-workflows

Workflows for Fission: Fast, reliable and lightweight function composition for serverless functions
Apache License 2.0
371 stars 42 forks source link

Define inline dynamic tasks using Javascript or Go #75

Open soamvasani opened 7 years ago

soamvasani commented 7 years ago

User should be able to define dynamic tasks (such as loops and other higher order functions) within a workflow definition, using Javascript or Go, and have them interpreted by the workflow engine as part of workflow execution.

To make dynamic task definition easy, the workflow engine should do most of the work of type checks etc. around dynamic task inputs and outputs. The user should have to write as little code as possible.

Both implementations (Javascript and Go) would involve the workflow engine loading the dynamic task and running it.

The Javascript implementation can use the existing Javascript interpreter, which already allows us to isolate functions as well as run them with timeouts. That will ensure that a misbehaved dynamic task from one workflow won't affect other workflows. If we solve #76 using Javascript, then that same set of primitives can be used within the dynamic task definition.

The Go implementation will need the builder to compile the Go tasks as one or more plugins. The workflow engine would load these as plugins and run them. It would have to do this in a separate process, for isolation; otherwise a misbehaving plugin could affect other workflows. There's a little more complexity in the Go implementation because of this out-of-process requirement. Also, the user would need to learn how to specify a workflow in Go using the workflow engine's type system, which is non-trivial because of some of the dynamic typing that the workflow engine does.

Overall, Javascript sounds like a better fit from a developer experience point of view, with the only advantage of the Go implementation being that you can probably get more performance out of it than the interpreted Javascript.