Action points and next steps

[huilisabrina]

Ordered by priority:

Essential steps for network updates:

• [x] Modify the S_flow function to allow one I node to infect multiple S nodes in the neighbors at one time step; (Inte) Done
• [x] Remove the num_h_seeds and num_s_seeds. Just need to have positive number of I nodes, but leave all others Susceptible (Inte) Done
• [x] Tweak parameters and set the "baseline" values (Inte) Done. The set-up of the graph - in particular, node degrees - matters a lot.
• [x] Integrate the serial codes with Inte's datasets (larger data size, cluster structure) (Hui) Done. See preprocess_network.py

A testing suite that generate:

• [ ] plotting the output (population overtime) (Camille)
• [ ] timing functions (runtime of the whole program runs) (Camille)
• [x] Modify the configuration of Spark output to surpress printing "INFO" messages. Note that the actual runtime of the program is much shorter than the actual time elapsed, due to the excessive printing. (Hui) Done. See comment below

Parallelization levels:

• [ ] Study/Cluster level: work within the "network_update" script to enable parallel network updating across disjoint subgraphs (Stephanie)
• [x] Monte Carlo level: a bash wrapper script that calls the "network_update" script, using the parameter values input to control each simulation run. (Hui) Done. See monte_carlo.bash and params_input.csv

What should be the output of our network updates script?

• A table of the nodes count for each state over time.
• Duration of the simulation.
• Parameter values.

Speed-up plots:

• local mode multi-cores speed-up pattern
• cluster mode multi-nodes multi-cores speed up pattern

Interpretations and verifying hypotheses: (relevant for what to put in params_input,csv)

• How to interpret the network data?
• What should we expect the results of the simulation to look like given the parameters?

[huilisabrina]

One note related to timing that @intekhab8 brought up during his testing, is that there are numerous INFO log messages that get printed our in the console. To avoid the excessive printing of the "INFO" messages, we can add the log level set up after setting up the Spark Context. I'll update the GF template file and all scripts using GF objects.

# Spark context
sc = SparkContext(conf=conf)

# surpress logging
sc.setLogLevel("ERROR")

After this, when running the simulation, only the beginning setup of the SparkEnv, Blockmanager INFO will be printed out, and all of the rest logging information will be suppressed.

In our final applications, i.e. network_update_monte_carlo.py , the log output will have the time elapsed calculated, so we can use the log file - parse our the time in the setence 2020/05/10 03:54:42 PM Simulation complete. Time elapsed: 34.72s - for timing comparison @beancamille :)