Repetitions parameter behaving strangely

[spoonerf]

When running the model with more than one repetition the model seems to only run about half the number of repetitions of the requested amount. For example if I set it at 6 it will run 3, if it's 12 it will run 5. My laptop should be able to run 6 at a time. Not an urgent problem just wondering if anyone else was experiencing this?

[nickmalleson]

Weird, it's almost certainly something I did. Will look at it next week

[spoonerf]

Hello,

Just updating with some information, fao @Sparrow0hawk:

OS: macOS Catalina 10.15.5 git branch: obesity git log: commit a5be7982204c3efcd3a0553f77e585a3fba0f6e7 command : python3 microsim/microsim_model.py -p model_parameters/default.yml

yaml (defaul.yml):

# Utility parameters for the microsimulation (python) part.
# These can also be passed as command line arguments
microsim:
  scenario: "BaseScenario" # name for scenario directory
  iterations: 100 # Number of model iterations. 0 means just run the initialisation.
  data-dir: devon_data # Root directory to load data from
  output: true # Whether to generate output or not
  output-every-iteration: false # Can optionally write output at every iteration, not just at the end
  debug: false # Whether to run some more expensive checks
  repetitions: 12 # How many times to run the model
  lockdown-file: "google_mobility_lockdown_daily.csv" # Read lockdown mobility data from a file? (Empty string means no lockdown)

# Parameters that are used to calibrate the model.
# These can only be passed using the parameters file
microsim_calibration:
  # These multiply the amount of hazard that individuals pass on to shops etc:
  hazard_individual_multipliers:
    presymptomatic: 1
    asymptomatic: 0.75
    symptomatic: 1.0
  # Different locations can also have their own multipliers (e.g. to make it easier to pass the disease on to shops than schools)
  hazard_location_multipliers:
    Retail: 1.0
    PrimarySchool: 1.0
    SecondarySchool: 1.0
    Home: 1.0
    Work: 1.0
  # This multiplies the risk received as individuals visit shops etc.
  risk_multiplier: 1.0

disease: # This is where the parameter for the disease model (the R part) can go (same format the 'microsim_calibration' above)
  current_risk_beta: 0.0175
  risk_cap: 5
  seed_days: 10
  exposed_dist: "weibull"
  exposed_mean: 2.56
  exposed_sd: 0.72
  presymp_dist: "weibull"
  presymp_mean: 2.3
  presymp_sd: 0.35
  infection_dist: "lognormal"
  infection_mean: 18
  infection_sd: 1.1
  asymp_rate: 0.4
  output_switch: TRUE
  rank_assign: FALSE
  obesity_40: 1.9
  obesity_35: 1.48
  obesity_30: 1.48
  overweight: 1
  cvd: 1
  diabetes: 1
  bloodpressure: 1

[spoonerf]

Also when the full number of iterations completes I get this error:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "microsim/microsim_model.py", line 1775, in _run_multicore
  File "microsim/microsim_model.py", line 1582, in run
  File "/Users/fionaspooner/Documents/ECEHH/COVID/RAMP-UA/microsim/r_interface.py", line 36, in __init__
    raise e
  File "/Users/fionaspooner/Documents/ECEHH/COVID/RAMP-UA/microsim/r_interface.py", line 28, in __init__
    R.initialize_r()
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/robjects/functions.py", line 198, in __call__
    .__call__(*args, **kwargs))
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/robjects/functions.py", line 125, in __call__
    res = super(Function, self).__call__(*new_args, **new_kwargs)
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/rinterface_lib/conversion.py", line 44, in _
    cdata = function(*args, **kwargs)
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/rinterface.py", line 621, in __call__
    raise embedded.RRuntimeError(_rinterface._geterrmessage())
rpy2.rinterface_lib.embedded.RRuntimeError: Error: Failed to install 'rampuaR' from GitHub:
  (converted from warning) installation of package ‘/var/folders/yc/bkg2k6pj6td2tpkj64v1yvbr0000gn/T//RtmpiC0nq4/file11200491b768b/rampuaR_2.1.0.tar.gz’ had non-zero exit status

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "microsim/microsim_model.py", line 1779, in <module>
    run_script()
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "microsim/microsim_model.py", line 1767, in run_script
    pool.starmap(_run_multicore, zip(models, iters,repnr))
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/multiprocessing/pool.py", line 276, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "microsim/microsim_model.py", line 1775, in _run_multicore
    return m.run(iter,rep)
  File "microsim/microsim_model.py", line 1582, in run
    self.r_int = RInterface(self.r_script_dir)
  File "/Users/fionaspooner/Documents/ECEHH/COVID/RAMP-UA/microsim/r_interface.py", line 36, in __init__
    raise e
  File "/Users/fionaspooner/Documents/ECEHH/COVID/RAMP-UA/microsim/r_interface.py", line 28, in __init__
    R.initialize_r()
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/robjects/functions.py", line 198, in __call__
    .__call__(*args, **kwargs))
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/robjects/functions.py", line 125, in __call__
    res = super(Function, self).__call__(*new_args, **new_kwargs)
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/rinterface_lib/conversion.py", line 44, in _
    cdata = function(*args, **kwargs)
  File "/Users/fionaspooner/opt/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/rpy2/rinterface.py", line 621, in __call__
    raise embedded.RRuntimeError(_rinterface._geterrmessage())
rpy2.rinterface_lib.embedded.RRuntimeError: Error: Failed to install 'rampuaR' from GitHub:
  (converted from warning) installation of package ‘/var/folders/yc/bkg2k6pj6td2tpkj64v1yvbr0000gn/T//RtmpiC0nq4/file11200491b768b/rampuaR_2.1.0.tar.gz’ had non-zero exit status

[spoonerf]

hmm doesn't seem to be a problem on master, will try merging obesity with master and see if that helps

[nnelissen]

Just FYI, my computer has been trying to run 6 repetitions (20 days) using code on master. Not exactly the same behaviour:

• It creates 5 directories, not 6 (numbered 0-4).
• I get m0 pickle files in each of these (initialisation works).
• It only completed 1 run (i.e. created the other pickle files), for the directory named '3'.

No errors. I'm wondering if there is an issue with starmap/multicore: it looks like it does one of the runs (randomly?) but 'drops' the others? Disclaimer, this computer (Ubuntu, i5-9400, 8GB RAM) has managed to 'hang' a lot (with and without memory errors) when trying to run the microsim, this may not be reproducible on another machine.

[nnelissen]

@nickmalleson For my computer, it seems to be related to multicore processing. If I replace the else statement from https://github.com/Urban-Analytics/RAMP-UA/blob/82559a89b4510bfc0a5b6796e69d07794dd6469d/microsim/microsim_model.py#L1751 onward with the code below (itertools didn't work, not sure why, didn't give an error just didn't do anything), it runs fine and produces the expected files for all runs

else:  # Run it multiple times in lots of cores
        m0 = Microsim(**msim_args)
        copyfile(parameters_file,os.path.join(m0.SCEN_DIR,"parameters.yml"))
        for r in range(repetitions):
            m = Microsim._make_a_copy(m0)
            m.run(iterations,r)

[nickmalleson]

Interesting, thanks @nnelissen, I'll keep looking at this. In the meantime I'm quite intrigued as why itertools.starmap didn't work as a replacement for pool.starmap. Could you put in some print statements into the Microsim.run() (https://github.com/Urban-Analytics/RAMP-UA/blob/82559a89b4510bfc0a5b6796e69d07794dd6469d/microsim/microsim_model.py#L1571) function to see how far it gets through that function once it is called by starmap?

[nickmalleson]

Just for clarity, the code works like this.

starmap runs the different models in different processes:

https://github.com/Urban-Analytics/RAMP-UA/blob/82559a89b4510bfc0a5b6796e69d07794dd6469d/microsim/microsim_model.py#L1767

It calls the _run_multicore function which simply takes a model as input and calls its run function (this would be unnecessary if python allowed you to use lambda functions with multiprocessing):

https://github.com/Urban-Analytics/RAMP-UA/blob/82559a89b4510bfc0a5b6796e69d07794dd6469d/microsim/microsim_model.py#L1774

The run function does the work for that model:

https://github.com/Urban-Analytics/RAMP-UA/blob/82559a89b4510bfc0a5b6796e69d07794dd6469d/microsim/microsim_model.py#L1571

[nnelissen]

@nickmalleson It executes itertools.starmap(_run_multicore, zip(models, iters,repnr)) but never calls _run_multicore

[nickmalleson]

Sorry, I can't replicate this on OS X, it works OK for me (just tested 9 repetitions of 10 iterations each).

[spoonerf]

I think this was just me installing the r package in a funny way