allow forking when running via nailgun

[cvogt]

Currently forking is limited to direct mode, because ProcessBuilder's inheritIO only works in direct mode. I experimented with not using inheritIO and using separate threads loop over reading the forked processes stderr and stdout and writing that to System.out and System.err.

One problem was that I had with separate threads is that when thread pools re-use threads, those threads will have captured the System.out and System.err at the time of creation (via a thread local), but nailgun changes that for every one of it's run threads, so those captured ones are no longer valid afterwards. The solution to that was passing the current nailgun threads System.out and System.err into the threads. This lead to the output being printed, but unlike with inheritIO, the lines and words printed to stderr and stdout end up being interleaved on a per character basis, not a per line basis, which is quite problematic.

We should check out what shells or terminals do with stdout and stderr in order to not show characters of the same lines interleaved, but always full lines.

[cvogt]

the fork feature limited to direct mode is here: https://github.com/cvogt/cbt/pull/457

[cvogt]

So turns out that the jdk buffers both stdout and stderr when you invoke a system process, if you don't use the INHERIT feature to bind it your own stdout/stderr file descriptors directly. Since this binding doesn't work for us when running via nailgun, because the stdout/stderr descriptors are those of the nailgun server process, no the client. So we can't use INHERIT and have to use PIPE instead, which means the jdk buffers. This is implemented in a method of UnixProcess and UnixProcess is final and uses native so it's not trivial to change this.

The problem with buffering those streams is that, even if the process is writing sequentially to one of them at a time, the order does get lost in the buffers and can't be recovered.

For a process that writes to both, this code demonstrates this nicely:

while( process.isAlive ){
  val ae = perr.available
  val ao = pout.available
  if(ae > 0 || ao > 0)
    println(ae,ao)
  if(ae > 0) {
    (1 to ae).foreach{ _ =>
      val c = perr.read
      if(c =!= -1){
        err.write(c)
        err.flush
      }
    }
  } else if(ao > 0) {
    (1 to ao).foreach{ _ =>
      val c = pout.read
      if(c =!= -1){
        out.write(c)
        out.flush
      }
    }
  }
}

The order the characters are written out is not the order in which the process sequentially wrote them.

[cvogt]

https://github.com/cvogt/cbt/pull/457 support this now with some caveats: lines may appear in different order. The order is maintained within each std out and std err, but not between them. But JDK's buffers make this impossible. Also there seems to be no way to react to System.in immediately except for blocking, which now requires a "Please press enter to continue..." after the process ended when in nailgun mode.