optimize many-time-steps case

[stevencarlislewalker]

Many time steps scale badly. Profiling with gprof suggest that expression evaluation is not the issue, but rather the saving of the simulation history. In particular the issue seems to be dynamic allocation of the object that stores the simulation history -- see attached picture from gprof.

[stevencarlislewalker]

More evidence for the hypothesis.

Experimental Setup

library(macpan2)
left = Sys.time()
sir = Compartmental(system.file("starter_models", "sir", package = "macpan2"))$simulators$tmb(
    time_steps = 2661 ## big = 2661, small = 2
  , state = c(S = 9999, I = 1, R = 0)
  , flow = c(foi = 0, gamma = 0.2)
  , beta = 0.4
  , N = 10000
)
sir$add$matrices(
  X = rbf(
     2661, ## big = 2661, small = 2
     100
  ),  
  b = rnorm(100),
  beta_values = empty_matrix,
  reset = empty_matrix
)
sir$insert$expressions(
    beta_values ~ -X %*% b
  , X ~ reset  ## comment this out when not using resetting
  , .phase = "before"
  , .at = Inf
)
.trash = sir$report(.phases = "during")
right = Sys.time()
right - left

Treatments and Results

## nothing big -- 0.6819479 s
## big X small T -- 0.834451 s
## small X big T -- 2.698912 s
## big X and T -- 30.11578 s 
## big X and T with resetting of X before the loop -- 3.078685 s

Observations

1. Big X has little impact on timing if the number of time steps, T, is small -- 0.83 is not much bigger than 0.68
2. Big T has a pretty big effect even if X is small -- 2.70 is ~4 times bigger than 0.68
3. When both X and T are big the slow down is huge -- 30.12!
4. But when X is reset by setting it to the empty matrix before the loop begins, the results for 'big X and T' are similar to those for 'small X big T' -- 3.08 is not much bigger than 2.70

Interpretation

These results and observations are consistent with the hypothesis that at each iteration, space in the history object is being made for a big X and then this space is filled only with an empty matrix because X is not in the mats_to_save.

[stevencarlislewalker]

This has been fixed by not repeatedly assigning empty matrices to the simulation history object.