Increasing trial duration

[MustafaCelen]

Hi, as you see below the duration after the 4th trial has increased a lot. I have been facing this issue a lot , I use the recommended weibull pdf hyperparameter ranges, only one channel I gave the max degree of freedom for all alpha, gamma, shape and scale. Is it normal to have such increase.

hyperparameters <- list(
  facebook_cost_alphas = c(0.5, 3),
  facebook_cost_gammas = c(0.3, 1),
  facebook_cost_shapes = c(0.0001, 10),
  facebook_cost_scales = c(0, 0.1),
  tv_cost_alphas = c(0.0001, 9.999),
  tv_cost_gammas = c(0.0001, 1),
  tv_cost_shapes = c(0.0001, 19.999),
  tv_cost_scales = c(0, 1),
  radio_cost_alphas = c(0.5, 3),
  radio_cost_gammas = c(0.3, 1),
  radio_cost_shapes = c(0.0001, 10),
  radio_cost_scales = c(0, 0.1),
  gads_display_cost_alphas = c(0.5, 3),
  gads_display_cost_gammas = c(0.3, 1),
  gads_display_cost_shapes = c(0.0001, 10),
  gads_display_cost_scales = c(0, 0.1),
  gads_video_cost_alphas = c(0.5, 3),
  gads_video_cost_gammas = c(0.3, 1),
  gads_video_cost_shapes = c(0.0001, 10),
  gads_video_cost_scales = c(0, 0.1),
  dv360_video_cost_alphas = c(0.5, 3),
  dv360_video_cost_gammas = c(0.3, 1),
  dv360_video_cost_shapes = c(0.0001, 10),
  dv360_video_cost_scales = c(0, 0.1),
  dv360_display_cost_alphas = c(0.5, 3),
  dv360_display_cost_gammas = c(0.3, 1),
  dv360_display_cost_shapes = c(0.0001, 10),
  dv360_display_cost_scales = c(0, 0.1),
  train_size = c(0.7,0.8)
)

 OutputModels <- robyn_run(
   InputCollect = InputCollect, # feed in all model specification
   #cores = 30, # NULL defaults to max available - 1
   ts_validation = TRUE,
   iterations = 15000, # 2000 recommended for the dummy dataset with no calibration
   trials = 10, # 5 recommended for the dummy dataset
   #add_penalty_factor = FALSE, # Experimental feature. Use with caution.
   outputs = FALSE # outputs = FALSE disables direct model output - robyn_outputs()
 )

Input data has 156 weeks in total: 2018-01-07 to 2020-12-25
Initial model is built on rolling window of 156 week: 2018-01-07 to 2020-12-25
Time-series validation with train_size range of 70%-80% of the data...
Using weibull_pdf adstocking with 30 hyperparameters (30 to iterate + 0 fixed) on 29 cores
>>> Starting 10 trials with 15000 iterations each using TwoPointsDE nevergrad algorithm...
  Running trial 1 of 10
  |================================================================================================================| 100%
  Finished in 21.04 mins
  Running trial 2 of 10
  |================================================================================================================| 100%
  Finished in 21.81 mins
  Running trial 3 of 10
  |================================================================================================================| 100%
  Finished in 28.73 mins
  Running trial 4 of 10
  |================================================================================================================| 100%
  Finished in 95.26 mins
  Running trial 5 of 10
  |================================================================================================================| 100%
  Finished in 95.65 mins
  Running trial 6 of 10
  |================================================================================================================| 100%
  Finished in 100.86 mins
  Running trial 7 of 10
  |================================================================================================================| 100%
  Finished in 102.58 mins
  Running trial 8 of 10
  |================================================================================================================| 100%
  Finished in 104.01 mins

[laresbernardo]

Thanks for reporting this @MustafaCelen

It def is an unexpected behaviour. I'd expect each trial to be quite similar (like the first too) and maybe increase timings given it has to append its results to the returned object. On our end, we can check wether we are reprocessing unnecessary information after each trial to reduce timings a bit, but not very confident this is happening. @Amyhaoming can you please double check for us?

The large increase in the 4th trial makes me believe there's something else running on that machine. @Amyhaoming can you try and replicate with 10 trials using the demo data please? Do we experience the same behaviour?

@MustafaCelen when analyzing your results, doesn't your models converge way before reaching the 15000 iterations per trial? Have you checked? Also, are the 10 trials really that different? I'd expect 5 trials to be OK though given they all start from a different random state and try to converge to similar results. Having 5 trials sounds good but I can't say for sure with your data. And yes, running Weibull takes a bit longer given it has an additional hyperparameter than Geometric, thus more flexibility but also slower.

[MustafaCelen]

Thank you for your reply @laresbernardo.

It was the first time I tried with 15000 iterations , I usually go with 10000 but the same problem appears. The VM is dedicated for me but i will double check with the system admins. For the trials my current best model was in the 7th trial , just to make sure I go with 10. NRMSE usually converges however Decomp rssd does not , when I finished the training with 15k iterations it was the firs time that decomp rssd converged. However not many improvements happened in the model.

[laresbernardo]

Please check this answer: https://www.facebook.com/groups/robynmmm/posts/1358750738226389 In short, DECOMP.RSSD doesn't necessarily has to converge.

[gufengzhou]

Please reopen if necessary.