Take inspiration from this?

coobas / wowp

WOWP (A WOrkfloW Framework in Python) is a modern, light-weight framework for integrated simulations in science.

http://pythonic.eu/wowp/

MIT License

2 stars 1 forks source link

Take inspiration from this? #1

Open coobas opened 9 years ago

coobas commented 9 years ago

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

http://nerds.airbnb.com/airflow/

Bitbucket: https://bitbucket.org/urbanj/wowp/issue/1

coobas commented 9 years ago

Original comment by Jan Pipek (Bitbucket: janpipek, GitHub: janpipek):

Something a bit different, but anyway: https://github.com/ContinuumIO/dask/

coobas commented 9 years ago

Original comment by Jakub Urban (Bitbucket: urbanj, GitHub: urbanj):

Being able to read/write one of the noflo languages (json or fbp) should enable using noflo-ui+flowhub.io for creating workflows (yet not running them)i.
Implmenting the protocol is probably a good idea for the future. There's a Python example in https://github.com/flowbased/protocol-examples to get started. Afterwards, we can fully exploit noflo-ui or create/find another gui that would communicate using this protocol (which is likely well designed and will be further developed and maintained).

coobas commented 9 years ago

Original comment by Jan Pipek (Bitbucket: janpipek, GitHub: janpipek):

Furthermore, we can take inspiration (even a negative one) from LabView and Windows Workflow Foundation (.NET).

coobas commented 9 years ago

Original comment by Jan Pipek (Bitbucket: janpipek, GitHub: janpipek):

At least this: http://noflojs.org/documentation/protocol/ should make us think :-)

coobas commented 9 years ago

Original comment by Jakub Urban (Bitbucket: urbanj, GitHub: urbanj):

What about NoFlo and its NoFlo-UI? Seems more than attractive B-)

coobas commented 9 years ago

Original comment by Jakub Urban (Bitbucket: urbanj, GitHub: urbanj):

Seems attractive!

“Configuration as code” is a principle we stand by for this purpose. While yaml or json job configuration would allow for any language to be used to generate Airflow pipelines, we felt that some fluidity gets lost in the translation. Being able to introspect code (ipython!, IDEs) subclass, meta-program and use import libraries to help write pipelines adds tremendous value. -- Absolutely agree.

+ We can fork their "Shiny UI"!