Suggestion: Python bindings for orchestrator operations

[m8pple]

A lot of the steps in the console feel more like commands in an EDA tool, which traditionally have been exposed as TCL, and more recently are exposed as python.

Binding some (or all) the commands to python functions would provide for a nice used experience (as they can use the REPL), and also provide decent scripting support.

Lua could work as well, but python is going to be the most user accessible for non-experts, e.g. undergraduate students.

[mvousden]

I don't understand this feature (though I vaguely recall that we spoke about it); here are some questions (which may not be relevant, but hey ho):

• Is the objective to provide a Python (Lua!) interface for potential operators?

• What objective of the Orchestrator does this satisfy?

• Does this mean maintaining the operator interface and a Python API?

• Would the Orchestrator have to host a listener that works with the Injector process to in/exfiltrate commands?

[m8pple]

Overall the operator interface feels like we are working in a custom REPL, with functions that can be called, parameters that can be passed, status that can be queried, and operations/results that might be passed back. So essentially it is an API, which could be driven by a human, or a batch file, or an intelligent script.

So my suggestion is that some of the orchestrator operations could be bound to a python class, e.g.:

class Orchestrator:
    def __init__(self, connection:str):
        ...

    def set_search_path(self, path:str):
       ...

    def get_tasks(self) -> Sequence[TaskInfo]:
        ....

    def load(self, name:str) -> TaskInfo:
        ...

    def deploy(self, task:TaskInfo):
        ....

Then it can either be driven interactively by the user in a python repl (with syntax that more people are likely to be familiar with), or used as part of a much more complicated script that mixes the generation of graphs, the execution with the orchestrator, and the capture/formatting of outputs. Ideally these things would throw on errors, and also act synchronously by default (so deploy didn't return until it was deployed).

An example would be the interface used by Gaith in his simulator, as it provides a good interface experience, while allowing for a lot of power.

It might not work so well with the asynchronous nature of the orchestrator where input and output seem to be decoupled, but there must be some notion of completion for a given call, otherwise how can we build and run multiple graphs in a sequence?

So to respond to your questions:

Is the objective to provide a Python (Lua!) interface for potential operators?

Yes, or at least the most important operations that will get used in scripts.

What objective of the Orchestrator does this satisfy?

• It makes it easier for humans to use, as they don't need to learn a new CLI grammar.
• You get a certain amount of tooling for free due to existing IDEs and REPLs that can provide auto-completion and doc-strings.
• It makes it easier to write robust scripts
• It makes it easier to write complicated scripts

Does this mean maintaining the operator interface and a Python API?

Possibly. It depends how useful the current operator interface is to people. It's also possible that lots of people love the current interface, I'm just suggesting something that I've seen in other tools.

Would the Orchestrator have to host a listener that works with the Injector process to in/exfiltrate commands?

That I don't know. I'm a little fuzzy on how to actually use the orchestrator synchronously, due to the decoupling of input and output - if a human is not watching, how do we know when each command has finished. Possibly batch files will solve this problem? Or the orchestrator is going to support something like futures or notifications to tell users when things have finished?

[mvousden]

Thanks for answering my questions; I think I've got more of a handle for what you're looking for. I see the value of this feature.

We could achieve this by having a Python script with "main" thread (T1), which spawns a thread (T2) to interact with the Orchestrator asynchronously over MPI. Specifically, T2 would communicate with the Injector process of the Orchestrator, which is responsible for Orchestrator command in/exfiltration. T2 would signal to T1 when a command is complete, errors, etc.

This overlaps somewhat with the scheduler functionality we want to add down the road; both of these components require exposing an interface to the Injector, which is how I propose we would start working on this feature.

Does this make sense, or am I missing the point?

[mvousden]

This is a really good idea. It is unlikely that we will actually get around to doing this as far as the POETS project is concerned, as it provides less utility than other more pressing concerns.