One of the problems with sos if that sos is built on top of python so there is a high probability of name clash. For example, users could accidentally define run and clash with our run action.
We have tried to resolve this issue with _input, SOS_CONFIG etc, but things like python still has a high risk of name conflict. I propose that we put everything into a sos namespace so that
sos.inputs is _input, the input for the current substep, which is the sos_target object.
sos.input is _input[0], which is the default for input with a single input. No automatic _input to _input[0] will be performed (which complicated the code quite a bit).
sos.group and sos.groups for results of group_by. In this way, group_by does not change _input by default.
sos.outputs and sos.output follows the same convension.
sos.step_input and sos.step_output are step_input and step_output.
sos.config is the SOS_CONFIG
sos.run_script("R", ...) is what R: will be translated to. The same goes to sos.python etc.. We may stick tosos.Retc, butsos.run` conflicts with the workflow execution function below.
sos.run will be sos_run for executing subworkflows.
sos.__dict__ will be sos_dict which is the namespace we use to store all runtime dictionaries.
In this way, sos adds on top of Python
Directives such as input:, output:, depends:, task:.
A syntax sugar for scripting running actions such as R: and Python:.
A sos namespace with data and functions.
Backward compatibility can be temporarily retained by making existing objects available (e.g. R:) but gives a deprecation warning.
Note that
the level of change may trigger the development of sos.2 in a separate repository, where we will reexamine the generic design and implementation issues with sos, such as remote/multi-node execution, and subworkflows.
we need to rigorously define these variables, how they are transferred (across steps, processes, and nodes), and how they change during the execution of the workflows. This is especially important for concurrent execution of substeps using async.
One of the problems with sos if that sos is built on top of python so there is a high probability of name clash. For example, users could accidentally define
runand clash with ourrunaction.We have tried to resolve this issue with
_input,SOS_CONFIGetc, but things likepythonstill has a high risk of name conflict. I propose that we put everything into asosnamespace so thatsos.inputsis_input, the input for the current substep, which is thesos_targetobject.sos.inputis_input[0], which is the default for input with a single input. No automatic_inputto_input[0]will be performed (which complicated the code quite a bit).sos.groupandsos.groupsfor results ofgroup_by. In this way, group_by does not change_inputby default.sos.outputsandsos.outputfollows the same convension.sos.step_inputandsos.step_outputarestep_inputandstep_output.sos.configis theSOS_CONFIGsos.run_script("R", ...)is whatR:will be translated to. The same goes tosos.pythonetc.. We may stick tosos.Retc, butsos.run` conflicts with the workflow execution function below.sos.runwill besos_runfor executing subworkflows.sos.__dict__will besos_dictwhich is the namespace we use to store all runtime dictionaries.In this way, sos adds on top of Python
input:,output:,depends:,task:.R:andPython:.sosnamespace with data and functions.Backward compatibility can be temporarily retained by making existing objects available (e.g.
R:) but gives a deprecation warning.Note that
sos.2in a separate repository, where we will reexamine the generic design and implementation issues withsos, such as remote/multi-node execution, and subworkflows.