populse / capsul

Collaborative Analysis Platform : Simple, Unifying, Lean
Other
7 stars 14 forks source link

inconsistency in Pipeline.get_linked_items() and dispatch_plugs() #286

Open denisri opened 1 year ago

denisri commented 1 year ago

Pipeline.get_linked_items() does not behave the same way when used on a regular (leaf) process or on a sub-pipeline inside the main pipeline. direction is not handled the same way:

                    if isinstance(node, Pipeline):
                        if in_outer_pipelines:
                            directions = ('links_from', 'links_to')
                        elif plug.output:
                            directions = ('links_from',)
                        else:
                            directions = ('links_to',)
                    elif plug.output:
                        directions = ('links_to',)
                    else:
                        directions = ('links_from',)

a terminal Process will thus yield its external links, while a sub-pipeline will provide only its internal links. So other nodes plugs connected to a sub-pipeline will not be reached from it. As a consequence, dispatch_plugs() and dispatch_value() miss some connections and actually do not dispatch to all nodes as it should. We could use in_outer_pipelines=True, but this will also propagate to other connected sub-pipeline, which is not always what we want. There is at least a "meaning" ambiguity on this method, it behaves differently on terminal processed and pipelines, differently depending on direction, in_sub_pipelines and in_outer_pipelines which makes its behaviour quite difficult to apprehend. And the doc currently says nothing about all this, so we don't really know what is the expected behaviour in each case.

denisri commented 1 year ago

We should maybe define clear use cases for this function: a. get processes/plugs connected to an outside plug of the main pipeline b. get processes/plugs inside a sub-pipeline connected to a sub-pipeline plug (maybe this is the same as case a. by using sub_pipeline.get_linked_items()) ?) c. get processes/plugs connected to a node plug in a pipeline (whatever its type), not going inside other sub-pipelines (sub-pipelines are treated as regular black-box processes here) d. get processes/plugs connected to a node plug in a pipeline (whatever its type), going inside other sub-pipelines

(do I forget other useful use cases ?) and provide the correct set of parameters go get the expected behaviour.