Closed sapetnioc closed 2 years ago
Being working on this issue, I need to make a link between configuration and execution context. More precisely, I need to know what are the datasets defined for a given execution context in order to be able to perform path generation (by the way, I propose to use path generation instead of completion it seems more clear to me). Even if the configuration is specific to a computing resource, the path generation must be available on client side. What I propose for this is to split ExecutionContext
in two parts:
PureExecutionContext
class that could be instantiated on client side to access to specific configuration resource configuration and perform operation such as path generation. This class could be created given the application configuration and a computing resource name. ExecutionContext
(extending PureExecutionContext
) that would only be instantiated on the computing resource side and would add specific resource such as a temporary folder. It would very likely be created given the JSON serialization of a PureExecutionContext
.I am not very found of PureExecutionContext
name but I have no better idea. It is inspired by the pathlib
module.
"Pure" is not very self-explanatory, indeed. We can imagine other names: "virtual" because it cannot run on the client, "remote" because it represents a remote engine, or "client" because it's the client part (which tells the contrary of the previous proposition, and thus I'm not fond of it). I'm not totally sure we really need to separate/distinguish them anyway ? As we already have an "engine" which tells about the remote status ? But I haven't thought about it.
Another option could be to have only one ExecutionContext
class and two functions to build an instance, one remote_execution_context(engine_name, configuration)
for client side and one execution_context(json_serialization)
for computing side. For the computing side, ExecutionContext
would be a context manager (with
statement) used to manage execution specific resource (temporary directory creation for instance). This would avoid to have to select another class name and would open the possibility to run a processes locally (in the client side) if ever we need it.
In new API, define a filenames completion feature for processes and pipelines.