eledhwen
commented
2 years ago Hi there 👋
From the onset a UI for designing and monitoring DAGs was indeed part of the vision for noleme-flow, now with that in mind, before we can get there there are two goals that must be attained:
- First, flow design: we need to consolidate the API (and runtime implementations) in order to maximize the number of options for reusable DAG designs. This is where the DAG morphology is decided and a potent API allowing maximum flexibility in matters of code reuse will go a long way in making noleme-flow DAGs easy to design and maintain, including via configuration and a hypothetical UI
- We are making ongoing progress on that front with the recently added Slices feature (sub-DAGs that can be manipulated like nodes), which also came with a step towards more generalized type abstractions for DAG nodes, going from the original dichotomy between "single-file" and "stream" modes, which enabled different capabilities, towards a generalized model enabling easier interconnexions (https://github.com/noleme/noleme-flow/pull/6, https://github.com/noleme/noleme-flow/pull/14).
- Second, connectors: we strongly believe that this type of technology requires a decently sized toolbox of connectors (plug-and-play sources, pipes and sinks) ready for use so you can kickstart the development of common pipeline patterns.
- Some work has been done in the
noleme-flow-connectorsrepository, but this is largely still in its infancy ; when the flow design features will be stabilized we expect to focus our attention here - In a not-so-distant future we expect to include connectors leveraging Jython and Janino for enabling more options with dynamic DAG node designs
- Some work has been done in the
Once those two goals have been attained, we should be in a great position for configuration-based DAG design using noleme-vault. And once that is accomplished, a UI is basically just a frontend for those configuration-based DAGs.
Out of curiosity, what is the application you would see this being useful for (both the DAG itself and a UI-based solution)?
Is there a plan to provide ui interface where users can design the DAG and edit the implementation code for each node?