cancellation_manager.pause_all_work() is insufficient to pause work

[Northbadge]

It is possible for work to continue running if no processes are registered on the cancellation manager when pause_all_work() is called.

Possible solutions: Also send SIGSTOP to the worker python process- however this has a race condition: pause_all_work -> new work scheduled -> pause python -> work is still being done. pausing python first would prevent pause_all_work from running so that's not an option.

Alternatively, have cancellation manager send SIGSTOP upon process registration when paused. This should work, I think, and is probably the easiest solution.

to verify: add a time_sleep right under .pause_children() in train_locally and look at the cpu graph

[mtrofin]

does this happen if we always make sure pause_all_work is followed by resume_all_work?

[Northbadge]

Yes- say the worker python thread is not waiting on a sub process clang, and at that moment pause all work is executed, no clangs are paused, then the worker is free to start sub process clangs going forward. Resume all work would either be a nop or send a SIGCONT for an already running process

[Northbadge]

randomly thought of something less race-y, so before I forget: we could have pause_all_work SIGSTOP itself while holding a process-creation-lock right before calling resume_all_work, so the parent process can "call" resume_all_work by sending a SIGCONT