Here's a slightly left-of-field idea: now that we have a stack machine that runs compile instructions, why not make running the binary one of those instructions?
This would involve something like:
making CompileExe push the output binary onto the files stack (ideally giving it a fresh name, so recipes can compile multiple binaries);
adding a new instruction, RunExe, which would pop a file from the stack, execute it, and either parse its results according to the backend or output it to a text file for a separate instruction to use;
ditching the current runner in favour of making recipes output RunExe instead.
Why would we do this?
1) Makes it easier to support recipes that don't need to run binaries later on (I'm thinking Herd recipes that just simulate the C, at this stage - in these, the lifter does all the work and the recipe would just need to parse the results from a file);
2) Makes the machine runner a bit more uniform, instead of being two separate concerns packaged into one binary;
3) Makes things like 'disable compilation' or 'disable running' generalise quite nicely; they'd instead become 'only dry run instructions of this kind', or maybe 'don't do compilation if the file requested already exists'.
This would need a fair bit of work in terms of the plan, as there's a distinction between compiler results and run results, but in the multi-stage world this probably needs to be cleaned up anyway, and I'll think about filing a separate issue.
A top-level worker pool allocates N goroutines containing interpreters for subjects.
Each interpreter advances through the recipe for the subject concurrently.
When a recipe needs to take a time-slice (for compiling or running), it synchronises with a scheduling goroutine that has a budget (probably also of N, maybe expressed as a semaphore). Actions such as running might take all of the budget, but compilations might just take one unit. This'd force running attempts to wait until a space is available for the run.
As such, compile and run jobs would just become specific types of generic 'job', with a generic scheduling system.
We might need to add additional synchronisation to deal with the nondeterministic chaos of all of the above, but it might just work.
Here's a slightly left-of-field idea: now that we have a stack machine that runs compile instructions, why not make running the binary one of those instructions?
This would involve something like:
CompileExe
push the output binary onto the files stack (ideally giving it a fresh name, so recipes can compile multiple binaries);RunExe
, which would pop a file from the stack, execute it, and either parse its results according to the backend or output it to a text file for a separate instruction to use;RunExe
instead.Why would we do this?
1) Makes it easier to support recipes that don't need to run binaries later on (I'm thinking Herd recipes that just simulate the C, at this stage - in these, the lifter does all the work and the recipe would just need to parse the results from a file); 2) Makes the machine runner a bit more uniform, instead of being two separate concerns packaged into one binary; 3) Makes things like 'disable compilation' or 'disable running' generalise quite nicely; they'd instead become 'only dry run instructions of this kind', or maybe 'don't do compilation if the file requested already exists'.
This would need a fair bit of work in terms of the plan, as there's a distinction between compiler results and run results, but in the multi-stage world this probably needs to be cleaned up anyway, and I'll think about filing a separate issue.