[RFC] Pipelining/1v1 match implementation idea

[seelengxd]

this looks kinda complicated idk if i am overthinking things (idea v1)

Phase 1: first convert testcase to use pipelining

Problem I can already see with this approach

• Lost testcase WA status (i.e. if result = False, set to WA...) that the previous hardcoded approach has. Runner becomes stupider unless I intentionally add another node there to set the status again to mimic the hardcoded approach of checking that node and changing status...

A step

is a node (proposed representation, every step now requires these properties)

{
  "id": 1,
  "inputs": [
    {
      "id": 1,
      "name": "params"
    }
  ],
  "outputs": [
    {
      "id": 1,
      "name": "status"
    },
    {
      "id": 2,
      "name": "stdout"
    },
    {
      "id": 3,
      "name": "stderr"
    }
  ]
}

run(inputs): outputs

A step should have a run method, that given the inputs (dict with all keys in inputs), return all keys of outputs.

A link

{
    id: 1,
    from_node_id: 1,
    from_socket_id: 2,
    to_node_id: 2,
    to_socket_id: 1
}

New steps

• Constant step
  • 0 inputs,
• Input step (special constant step actually)
  • Only 1, 0 inputs, any number of outputs (>= 1)
• Output step
  • Only 1, 0 outputs, any number of inputs (>= 1)

Rough idea of how I think the node runner should work

on termination, the runner can start spawning nodes to form, including a copy of the current_node (which i think is necessary for a loop node)

Phase 2: implementation of a match

with this, a match could still be a programming task

New node

• loop_step
  • definitely the most cursed one implementation wise
  • unlike other nodes that just activate all its links, this node programmatically activates one link

Future improvements

(i.e. there are some flaws but this looks complicated enough already (maybe we are cooked))

• Loop node doesn't actually need output/status on the first interation

[shenyih0ng]

We can probably shift the responsibility of setting the status (e.g. WA) back to the ProgrammingTask which takes the result of running the test case and compares it with expected test case result. The intuition is that the runner that runs the test case do not have to know what exactly is the status, instead it just runs the compute graph.

In our test data we are implicitly assuming that MatchStringStep checks for a correct match, but now I am leaning towards it simply be a step that computes whether it is a match or not. Subsequently, this match result will be passed into another node that accumulates the previous output and computes a final result that represents the test case result. With this we can do test cases that accepts (Status.Ok) non-matches. It is same as your suggestion for adding a extra node but I think this gives us more flexibility while keeping runner simple.