jiqicn
commented
2 years ago Pipeline Design
General principles
- Use built-in nodes if possible.
- Annotations and labels are beneficial.
Different ways of modularization in KNIME
Metanodes
The simplest way to clean up messy workflows, and that is the only thing that metanodes can do: metanotes are purely used to organize your workflows better. Also note that metanodes created by users can't be uploaded to KNIME Hub.
Components
Components are nodes that contain a sub-workflow, which lets you bundle functionality for sharing and reusing. Components are similar to metanodes but with more functions.
To refer:
1. A component can encapsulate flow variables
Flow variables created inside components will not leave the component unless this is expressly set as an component output, and verse vice, flow variables created in the workflow but outside of the component will not enter the component, unless expressly set as an component input.
2. Component configuration through configuration window and nodes.
A component has a configuration window to setup itself. Also, this window can be customized to some extend by adding configuration notes.
3. Only a component is given a view (dashboard)
The configuration, widget, and visualization nods inside a components can be organized and displayed within a single view (dashboard) with interactions. Layout is also customizable.
4. Components can be uploaded to KNIME Hub for sharing
In this way, components can be published for collaboration and reusage. There will be a webpage of the component that can show descriptions of functionalities, input/output, and links to sample workflows that demonstrate the usage of the component.
Workflow Services
In KNIME, it's also possible to create workflow as a service that can be called by other workflows. This is done by using the KNIME workflow services set of nodes.
Different from using components which copys all the nodes inside and the configuration to the target workflow, a workflow service is merely referenced. The caller workflow gets to know what to call from the callee workflows, but the exact nodes that will be executed where it is located, and this is hidden to the caller workflow.
Comparing to components, workflow services has the following pros and cons:
Advantages
- Workflows will be smaller since it contains fewer nodes, so faster when loading.
- It's easier to test a workflow than a component.
- Since multiple workflow services can be triggered at the same time, it may scale up the cases that are computationally intensive.
- Through workflow services, it's also possible to offload compute intensive tasks to a KNIME server (commercial feature).
- Data only available from server may be exposed to clients by a workflow service.
Disadvantages
- Workflow services can be changed without noticing at all. When using shared components, users can decide whether they want to update them or not if something has changed in the referenced component. This is not possible with workflow services.
To summarize, workflow services are good when splitting a big task into smaller sub tasks that can be performed individually (e.g. collecting data, preparing data, training machine learning models, comparing performance, etc.), while components are more like nodes (with configuration and view) that are highly modularized.
To refer:
This issue is collecting random ideas for designing the KNIME workflows in our project. so it might not be organized well (can be summarized as a separate document in the future if needed).