dspinellis / dgsh

Shell supporting pipelines to and from multiple processes
http://www.spinellis.gr/sw/dgsh/
Other
323 stars 23 forks source link

Incorrect graph in nested dgsh invocations #29

Closed dspinellis closed 7 years ago

dspinellis commented 7 years ago

When running example/5x5.sh with DGSH_DOT_DRAW the following file is created.

digraph {
        n0 [label="bash --dgsh-negotiate -c matrix"];
        n0 -> n1;
        n1 [label="scatter-gather"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31690"];
        n0 -> n1;
        n1 [label="scatter-gather"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31726"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31732"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31728"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31729"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31730"];
        n0 -> n1;
        n1 [label="paste"];
}
mfragkoulis commented 7 years ago

Good news and bad news: The good news is the graph is correct. This is evident from the explanation given below.

The bad news is there are three sequences of negotiation going on (it wasn't clear to me earlier).

  1. call matrix negotiates with cat; the negotiation happens once and leads to the execution of the matrix function.
  2. In the matrix function dgsh-parallel negotiates with cat; the negotiation happens once and leads to the 5-degree parallel execution of the row function
  3. In the row function dgsh-parallel negotiates with paste; the negotiation will happen once for each execution of the function, that is five times.

By counting the distinct negotiation processes we find seven distinct graphs as per the contents of the dot file.

call is another name for dgsh --dgsh-negotiate. This breaks the graph into pieces.

dspinellis commented 7 years ago

Let's discuss this, so that I fully understand it.

dspinellis commented 7 years ago

We agreed to avoid invoking the dgsh-parallel script with dgsh --dgsh-negotiate. Instead, scripts will inherit DGSH_IN and DGSH_OUT from the called process.

dspinellis commented 7 years ago

I removed the--dgsh-negotiate from dgsh-parallel. Now example/5x5 is no longer running.