Start jobs in their own processes

[rjoleary]

Each jobs has its own process. This has the following benefits:

• Failure isolation: If a test panics unexpectedly (for example it runs out of memory), only that test is affected and not the whole system.
• Cancel jobs: Allow jobs to be force cancelled simply by killing the process.
• Resource limits: Each job can have its own limits on memory, fds, cpu, etc...

[insomniacslk]

Thanks @rjoleary ! This is something @marcoguerri and I have considered in the past, for the reasons you have mentioned above. The current design is also at risk of goroutine leak, and using processes for jobs will alleviate that too. I am wondering if we should have both, or just go with processes for everything

[xaionaro]

I am wondering if we should have both, or just go with processes for everything

Just to be on the same page: The kind-of-only way to have a process for each job in Go is to use os/exec (Go does not allow just to fork()/clone()). So it would require to essentially redesign some things within ConTest. So it will cost much more to support both "job-as-goroutine" and "job-as-process" (they will have different kinds of "manager->worker" communications).

Therefore it seems to me that if we want to do job-as-process then we need to drop support of job-as-goroutine. And if we afraid of complicating ConTest for simple users with simple cases, then we may make it as transparent as possible: to implement both functions (job management and job execution) within one binary which just calls itself (via os/exec) for job execution.