ICRAR / daliuge

The DALiuGE Execution Engine
GNU Lesser General Public License v2.1
24 stars 7 forks source link

Named ports #192

Closed awicenec closed 2 years ago

awicenec commented 2 years ago

Completed implementation of support for named ports. This now includes support for docker and bash components as well as the previously already supported pyfunc components. The common parts of the identification and mapping of the named ports has been moved into droputils.py. The change is backwards compatible in that it is still possible to use the command line replacement placeholders. However, there is one known side-effect which will break graphs in rare cases when they are modified using current versions of EAGLE with the named ports extension:

Mixing old and new ports: An old graph will fail, if an existing component is upgraded with a new port AND there had already been un-named ports on the same side (input or output). Work-around: All ports of that component should be connected again.

Command lines are constructed from the following parts (and in this order}:

{command} {argumentString} {command_line_arguments}

where {command} and {command_line_arguments} are defined by the respective component arguments. {argumentString} is constructed from positional and kw arguments defined as application arguments in EAGLE and augmented by ports with matching names, if defined. That means that values are read from the ports rather than from the respective argument definitions. The definition of arguments as positional is also used for port values. Arguments, which have undefined values (including zero length and white space strings and boolean False) are removed from the list, after the port matching is performed. Ports, which can't be mapped to a defined argument are not put on the command line. That feature also allows to deal with situations where the application produces outputs as a side effect (e.g. untar), but the flow needs to continue.

The implementation in principle allows mixing of a command line containing the %iX %oX placeholders, but will often lead to wrong command lines. We will replace them with new, named placeholders, which will then allow to put arguments (including ports) by name on the command line.

coveralls commented 2 years ago

Coverage Status

Coverage decreased (-0.04%) to 82.196% when pulling 9f93047e0c65fe1beb79c4cebfc932832dd7c15e on named-ports into a92ee81d5657deed33ba126127db3918a21e3fe6 on master.

awicenec commented 2 years ago

Thanks Nicholas, wrt, Documentation: Yes, absolutely! Tests: They are implicitly done already, since I have changed a few of the graphs, which are being run in the automatic tests. Maybe adding a few additional negative unit tests would be a good idea. We need to be able to catch quite a number of cases, which are incorrect.