Changes data/run.py to use multiprocessing library instead of bash

[nkremerh]

Two additional pieces:

Is it possible to replicate the behavior of the bash script more closely? Key difference: the job update interval is now being used to print out progress on submitting jobs rather than printing how many jobs are left at the end. The preferred outcome is the latter, with every submitted job printed and a countdown provided at the end.

When sending a termination signal (CTRL + c), it would be preferable to not see a huge wave of Python KeyboardInterrupt runtime errors. There are ways to handle either the stdout and stderr of a program (such as sending it to /dev/null which may be too UNIX-like for Windows environments) or to catch the termination signal and handle it more gracefully.

[nkremerh]

Closer. The currently running simulations should terminate when interrupted.

[colinhanrahan]

Closer. The currently running simulations should terminate when interrupted.

pool.terminate() terminates all processes in the pool. Also, the jobs remaining countdown added some unpleasant complexity. You have to share variables across processes.

[nkremerh]

It should, but it does not.

Closer. The currently running simulations should terminate when interrupted.

pool.terminate() terminates all processes in the pool. Also, the jobs remaining countdown added some unpleasant complexity. You have to share variables across processes.

It should, but it does not.

[colinhanrahan]

@nkremerh How about now? Edit: looks like it didn't work from my testing. Should we be using multithreading or multiprocessing?

[nkremerh]

Let me think on it. Since you're about to take on bigger Kanban cards, let me take ownership of this one. Please keep this branch alive while I work on getting it touched up and merged. The core logic is there, so there are likely just a few architectural changes I need to make.

[colinhanrahan]

The impression I'm getting is that os.system is spawning its own subprocesses that aren't tracked by the Pool, so you'll need to keep track of them individually to terminate them. I was looking into subprocess.Popen and ThreadPoolExecutor if you're interested.

[nkremerh]

That would make sense. Good intuition!

[nkremerh]

@colinhanrahan Please review.

A note: interrupting the jobs kills all the processes in the pool now, and only the main process (run.py) complains. This is an acceptable compromise for me as similar behavior happens if you terminate the GUI window early and other edge cases of interrupting the simulation.

[colinhanrahan]

@nkremerh

• The while len(results) > 0: loop is busy-waiting. You could add a short sleep, but if enough simulations are run in parallel, there's a chance that some job updates will be skipped.
• ~With these changes, I believe Bash can now be removed from the config, README, etc.~ fixed while reviewing.
• numParallelSimJobs can be set arbitrarily high. Creating all the pool jobs at once doesn't cause any immediate issues, but we might want to cap the number of parallel jobs at os.cpu_count() just in case.

Other things I found:

• This is a nitpick, but the configs generated by data collection are difficult to read since all options are on one line. We could lump in a fix as part of this feature.
• There's a bug left over from adding multiple decision models that prevents incomplete logs from being rerun. On line 66, we're comparing the string model to the lists stored in dataOpts["decisionModels"]. We should be using the same naming logic seen on lines 22-23.

[nkremerh]

I originally included a sleep in the busy waiting, but that led to job count updates getting skipped (as you noted).

The one-liner configs are minified for file size reasons. I'm fine with them not being pretty printed.

Fixed the bug by simply trusting if the user leaves unfinished jobs in the data directory and runs make data, then they want those jobs run.

[colinhanrahan]

I pushed some small changes, but otherwise looks good. Let me know if I broke anything.