There are a few problems/questions we will need to solve to implement custom nodes. This is intended as a way to frame discussion about potential approaches, some of which I may have missed in this initial pro/con list. Some of these approaches might also become more/less difficult as we containerize the application.
1) Where do custom nodes live?
pyworkflow/custom_nodes
Pro: This approach provides an easy way for custom nodes to import the Node class to extend.
Workflow's root_dir (defaults to /tmp)
Pro: This approach could provide an easy way for moving all Workflow data to another machine. I.e., the root_dir contains the workflow, custom nodes, intermediate execution data, and any output files.
Con: Custom nodes would need a way to import the Node class.
Other directory: We probably do not want to consider this option.
2) How does a user add a custom node/how to install additional packages?
PR #66 goes with a "user uploads a file to the server" approach. This facilitates a UI-based approach similar to uploading CSV files for an individual node. If the custom node only includes packages pre-installed (like pandas), this could be a seamless approach where the node listing refreshes and the user can drag-and-drop the new node instantly.
Another approach to consider could be prompting the user to add their node to the Pipfile, run pipenv install to load in any packages, and (re)start the server for changes to take effect. This is closer to the approach we have discussed.
Benefits to this approach include:
Installs any required packages
Potentially handles where the node is (problem number 1 above).
It seems a majority of users would be comfortable with a non-UI approach, and one that is more flexible. This should accommodate that use case
Potential downsides:
Would the user have to "package-ify" their custom node to add it to the Pipfile in order to add it? This is not as user-friendly as just writing a single .py file that defines class MyNode(Node):.
There are a few problems/questions we will need to solve to implement custom nodes. This is intended as a way to frame discussion about potential approaches, some of which I may have missed in this initial pro/con list. Some of these approaches might also become more/less difficult as we containerize the application.
1) Where do custom nodes live?
pyworkflow/custom_nodesNodeclass to extend.root_dir(defaults to/tmp)root_dircontains the workflow, custom nodes, intermediate execution data, and any output files.Nodeclass.2) How does a user add a custom node/how to install additional packages?
PR #66 goes with a "user uploads a file to the server" approach. This facilitates a UI-based approach similar to uploading CSV files for an individual node. If the custom node only includes packages pre-installed (like pandas), this could be a seamless approach where the node listing refreshes and the user can drag-and-drop the new node instantly.
Another approach to consider could be prompting the user to add their node to the
Pipfile, runpipenv installto load in any packages, and (re)start the server for changes to take effect. This is closer to the approach we have discussed.Benefits to this approach include:
Potential downsides:
Pipfilein order to add it? This is not as user-friendly as just writing a single .py file that definesclass MyNode(Node):.