search

facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.13k stars 336 forks source link

Different Results in Recreated Model #800

Open ddempseyr opened 1 year ago

ddempseyr commented 1 year ago

Project Robyn

Describe issue

I believe that I've found an edge case that causes the output (e.g. coefficients, xDecompAgg) from robyn_recreate to drastically differ from the original solution. Our team noticed this in our production environment and is causing unexpected results in our budget allocation and refresh jobs.

To confirm the issue was not in our pipeline, we did some testing and were able to replicate the issue using the data from demo script as well. Based on our findings, the issue (appears) to only occur when the window start date is reduced (e.g. from 2016-01-01 to 2017-01-01) and adstock is changed from "geometric" to "weibull_pdf."

See the difference in the original xDecompAgg and recreated xDecompAgg from the reproducible example below.

Screen Shot 2023-08-10 at 5 22 35 PM

Provide reproducible example

library(Robyn)
Sys.setenv(R_FUTURE_FORK_ENABLE = "true")
options(future.fork.enable = TRUE)
create_files <- TRUE

# ------------------------------------- Load Data ------------------------------------------------- #

data("dt_simulated_weekly")
head(dt_simulated_weekly)

data("dt_prophet_holidays")
head(dt_prophet_holidays)

robyn_directory <- "~/Desktop"

# ------------------------------------- Create Inputs ------------------------------------------------- #

adstock <- 'weibull_pdf'
window_start <- '2017-01-01'
InputCollect <- robyn_inputs(
  dt_input = dt_simulated_weekly,
  dt_holidays = dt_prophet_holidays,
  date_var = "DATE", # date format must be "2020-01-01"
  dep_var = "revenue", # there should be only one dependent variable
  dep_var_type = "revenue", # "revenue" (ROI) or "conversion" (CPA)
  prophet_vars = c("trend", "season", "holiday"), # "trend","season", "weekday" & "holiday"
  prophet_country = "DE", # input country code. Check: dt_prophet_holidays
  context_vars = c("competitor_sales_B", "events"), # e.g. competitors, discount, unemployment etc
  paid_media_spends = c("tv_S", "ooh_S", "print_S", "facebook_S", "search_S"), # mandatory input
  paid_media_vars = c("tv_S", "ooh_S", "print_S", "facebook_I", "search_clicks_P"), # mandatory.
  # paid_media_vars must have same order as paid_media_spends. Use media exposure metrics like
  # impressions, GRP etc. If not applicable, use spend instead.
  organic_vars = "newsletter", # marketing activity without media spend
  # factor_vars = c("events"), # force variables in context_vars or organic_vars to be categorical
  window_start = window_start, # used to be 2016-01-01
  window_end = "2018-12-31",
  adstock = adstock # geometric, weibull_cdf or weibull_pdf.
)
print(InputCollect)

hyper_names(adstock = InputCollect$adstock, all_media = InputCollect$all_media)
plot_adstock(plot = FALSE)
plot_saturation(plot = FALSE)
hyper_limits()

# Example hyperparameters ranges for Geometric adstock
hyperparameters <- list(
  facebook_S_alphas = c(0.5, 3),
  facebook_S_gammas = c(0.3, 1),
  facebook_S_shapes = c(0, 10),
  facebook_S_scales = c(0, 0.1),
  print_S_alphas = c(0.5, 3),
  print_S_gammas = c(0.3, 1),
  print_S_shapes = c(0, 10),
  print_S_scales = c(0, 0.1),
  tv_S_alphas = c(0.5, 3),
  tv_S_gammas = c(0.3, 1),
  tv_S_shapes = c(0, 10),
  tv_S_scales = c(0, 0.1),
  search_S_alphas = c(0.5, 3),
  search_S_gammas = c(0.3, 1),
  search_S_shapes = c(0, 10),
  search_S_scales = c(0, 0.1),
  ooh_S_alphas = c(0.5, 3),
  ooh_S_gammas = c(0.3, 1),
  ooh_S_shapes = c(0, 10),
  ooh_S_scales = c(0, 0.1),
  newsletter_alphas = c(0.5, 3),
  newsletter_gammas = c(0.3, 1),
  newsletter_shapes = c(0, 10),
  newsletter_scales = c(0, 0.1),
  train_size = c(0.5, 0.8)
)
InputCollect <- robyn_inputs(InputCollect = InputCollect, hyperparameters = hyperparameters)
print(InputCollect)

# ------------------------------------- Create Model ------------------------------------------------- #

OutputModels <- robyn_run(
  InputCollect = InputCollect, # feed in all model specification
  cores = NULL, # NULL defaults to (max available - 1)
  iterations = 500, # 2000 recommended for the dummy dataset with no calibration
  trials = 5, # 5 recommended for the dummy dataset
  ts_validation = TRUE, # 3-way-split time series for NRMSE validation.
  add_penalty_factor = FALSE # Experimental feature. Use with caution.
)
print(OutputModels)

OutputCollect <- robyn_outputs(
  InputCollect, 
  OutputModels,
  pareto_fronts = "auto", # automatically pick how many pareto-fronts to fill min_candidates (100)
  # min_candidates = 100, # top pareto models for clustering. Default to 100
  # calibration_constraint = 0.1, # range c(0.01, 0.1) & default at 0.1
  csv_out = "pareto", # "pareto", "all", or NULL (for none)
  clusters = TRUE, # Set to TRUE to cluster similar models by ROAS. See ?robyn_clusters
  export = create_files, # this will create files locally
  plot_folder = robyn_directory, # path for plots exports and files creation
  plot_pareto = create_files # Set to FALSE to deactivate plotting and saving model one-pagers
)
print(OutputCollect)

# ------------------------------------- Export Model ------------------------------------------------- #

select_model <- "2_34_11" # Pick one of the models from OutputCollect to proceed
ExportedModel <- robyn_write(InputCollect, OutputCollect, select_model, export = create_files)
print(ExportedModel)
print(names(ExportedModel$ExportedModel$plot_folder))

# ----------------------------------- Read and Recreate Model ---------------------------------------- #

robyn_write(InputCollect, OutputCollect, select_model)

json_file <- paste(ExportedModel$ExportedModel$plot_folder, "RobynModel-", select_model, ".json", sep="")
json_data <- robyn_read(json_file)
print(json_data)

RobynRecreated <- robyn_recreate(
  json_file = json_file,
  dt_input = dt_simulated_weekly,
  dt_holidays = dt_prophet_holidays,
  quiet = FALSE)
InputCollectX <- RobynRecreated$InputCollect
OutputCollectX <- RobynRecreated$OutputCollect

# ----------------------------------- Compare Original and Export Model ---------------------------------------- #
library('dplyr')
filter(OutputCollect$xDecompAgg, solID == select_model)
filter(OutputCollectX$xDecompAgg, solID == select_model)

Environment & Robyn version

> packageVersion("Robyn")
[1] ‘3.10.3’
> sessionInfo()
R version 4.2.0 (2022-04-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.7.6

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] dplyr_1.1.2    doRNG_1.8.6    rngtools_1.5.2 foreach_1.5.2  Robyn_3.10.3   arrow_8.0.0   

loaded via a namespace (and not attached):
 [1] nlme_3.1-157        bitops_1.0-7        matrixStats_1.0.0   lubridate_1.9.2     bit64_4.0.5         RColorBrewer_1.1-3  doParallel_1.0.17   httr_1.4.6          rprojroot_2.0.3     rstan_2.21.8       
[11] tools_4.2.0         utf8_1.2.3          R6_2.5.1            rpart_4.1.16        mgcv_1.8-40         colorspace_2.1-0    withr_2.5.0         tidyselect_1.2.0    gridExtra_2.3       prettyunits_1.1.1  
[21] processx_3.8.2      bit_4.0.4           compiler_4.2.0      glmnet_4.1-6        textshaping_0.3.6   cli_3.6.1           rvest_1.0.3         xml2_1.3.3          labeling_0.4.2      scales_1.2.1       
[31] ggridges_0.5.4      callr_3.7.3         systemfonts_1.0.4   stringr_1.5.0       digest_0.6.32       StanHeaders_2.26.27 extraDistr_1.9.1    pkgconfig_2.0.3     rlang_1.1.1         rstudioapi_0.13    
[41] shape_1.4.6         prophet_1.0         generics_0.1.3      farver_2.1.1        jsonlite_1.8.7      zip_2.3.0           inline_0.3.19       RCurl_1.98-1.10     magrittr_2.0.3      loo_2.6.0          
[51] patchwork_1.1.2     Matrix_1.5-3        Rcpp_1.0.10         munsell_0.5.0       fansi_1.0.4         reticulate_1.30     lifecycle_1.0.3     stringi_1.7.12      pROC_1.18.2         yaml_2.3.7         
[61] pkgbuild_1.4.2      plyr_1.8.8          grid_4.2.0          parallel_4.2.0      crayon_1.5.2        lattice_0.20-45     splines_4.2.0       lares_5.2.0         ps_1.7.5            pillar_1.9.0       
[71] codetools_0.2-18    stats4_4.2.0        glue_1.6.2          rpart.plot_3.1.1    RcppParallel_5.1.7  png_0.1-8           vctrs_0.6.3         nloptr_2.0.3        gtable_0.3.3        purrr_1.0.1        
[81] tidyr_1.3.0         assertthat_0.2.1    ggplot2_3.4.2       openxlsx_4.2.5.2    h2o_3.38.0.1        ragg_1.2.2          survival_3.3-1      minpack.lm_1.2-3    tibble_3.2.1        iterators_1.0.14   
[91] timechange_0.2.0    here_1.0.1
gufengzhou commented 1 year ago

Sorry for the delay. In #798 that's related to refreshing, the user has also observed a difference between original and recreated model. I've identified to cause to be the rounding issue of hyperparameters when creating the model json file. I'll implement the fix asap and please let me know if it improves.

ddempseyr commented 1 year ago

@gufengzhou Sorry for the delay! Thanks so much for looking into this and committing a fix. I'm hoping to test the fix this week and follow up with the results.

ddempseyr commented 1 year ago

@gufengzhou Sorry again for my slow response. I wanted to follow up and let you know that we should have the results of our test sent over by end of day tomorrow.

ddempseyr commented 1 year ago

@gufengzhou Great news. After installing Robyn locked to commit 38df2237943a45a9001bd05904d392d6a96adfc3, I re-ran the reproducible example above, and confirm that the results from the original model and recreated model are one-to-one.

Next, I am going to run the same test using our customers data since the discrepancy was larger than the discrepancy seen using the demo data. Unfortunately, we will not be able to provide these files but I will follow up with the results.

Thanks again for your help.

ddempseyr commented 1 year ago

@gufengzhou We rebuilt our docker images pointed to the commit for the fix, reran our pipelines, and are still seeing relatively large discrepancies on our customer's recreated models. Unfortunately, we cannot provide the training dataset required to recreate the model because it would expose our customer's data.

Are there any other reasons that you can think of that would cause the discrepancy or are any other Robyn users experiencing similar issues?

ddempseyr commented 1 year ago

@gufengzhou I was able to anonymize our data so that it can be sent over. See the script below and the three files required to recreate the model.

Instructions to replicate the issue:

train.csv events.csv

{
  "InputCollect": {
    "date_var": ["report_date"],
    "dayInterval": [1],
    "intervalType": ["day"],
    "dep_var": ["dependent__sales__none__none__none"],
    "dep_var_type": ["revenue"],
    "prophet_vars": ["trend", "season", "weekday", "holiday"],
    "prophet_signs": ["default", "default", "default", "default"],
    "prophet_country": ["US"],
    "context_vars": ["promotions_a", "promotions_b", "promotions_c", "subscriptions_a"],
    "context_signs": ["default", "default", "default", "default"],
    "paid_media_vars": ["paid_media_exposure_a", "paid_media_exposure_b", "paid_media_exposure_c", "paid_media_exposure_d", "paid_media_exposure_e", "paid_media_exposure_f", "paid_media_exposure_g", "paid_media_exposure_h", "paid_media_exposure_i", "paid_media_exposure_j", "paid_media_exposure_k"],
    "paid_media_signs": ["positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive"],
    "paid_media_spends": ["paid_media_spend_a", "paid_media_spend_b", "paid_media_spend_c", "paid_media_spend_d", "paid_media_spend_e", "paid_media_spend_f", "paid_media_spend_g", "paid_media_spend_h", "paid_media_spend_i", "paid_media_spend_j", "paid_media_spend_k"],
    "mediaVarCount": [11],
    "exposure_vars": ["paid_media_exposure_a", "paid_media_exposure_b", "paid_media_exposure_c", "paid_media_exposure_d", "paid_media_exposure_e", "paid_media_exposure_f", "paid_media_exposure_g", "paid_media_exposure_h", "paid_media_exposure_i", "paid_media_exposure_j", "paid_media_exposure_k"],
    "all_media": ["paid_media_spend_a", "paid_media_spend_b", "paid_media_spend_c", "paid_media_spend_d", "paid_media_spend_e", "paid_media_spend_f", "paid_media_spend_g", "paid_media_spend_h", "paid_media_spend_i", "paid_media_spend_j", "paid_media_spend_k"],
    "all_ind_vars": ["trend", "season", "weekday", "holiday", "promotions_a", "promotions_b", "promotions_c", "subscriptions_a", "paid_media_spend_a", "paid_media_spend_b", "paid_media_spend_c", "paid_media_spend_d", "paid_media_spend_e", "paid_media_spend_f", "paid_media_spend_g", "paid_media_spend_h", "paid_media_spend_i", "paid_media_spend_j", "paid_media_spend_k"],
    "unused_vars": [],
    "window_start": ["2021-09-03"],
    "rollingWindowStartWhich": [1],
    "window_end": ["2023-09-19"],
    "rollingWindowEndWhich": [747],
    "rollingWindowLength": [747],
    "refreshAddedStart": ["2021-09-03"],
    "adstock": ["weibull_pdf"],
    "hyperparameters": {
      "paid_media_spend_a_alphas": [0.5, 3],
      "paid_media_spend_a_gammas": [0.3, 1],
      "paid_media_spend_a_scales": [0.0001, 0.1],
      "paid_media_spend_a_shapes": [0, 2],
      "paid_media_spend_b_alphas": [0.5, 3],
      "paid_media_spend_b_gammas": [0.3, 1],
      "paid_media_spend_b_scales": [0.0001, 0.1],
      "paid_media_spend_b_shapes": [0, 2],
      "paid_media_spend_c_alphas": [0.5, 3],
      "paid_media_spend_c_gammas": [0.3, 1],
      "paid_media_spend_c_scales": [0.0001, 0.1],
      "paid_media_spend_c_shapes": [0, 2],
      "paid_media_spend_d_alphas": [0.5, 3],
      "paid_media_spend_d_gammas": [0.3, 1],
      "paid_media_spend_d_scales": [0.0001, 0.1],
      "paid_media_spend_d_shapes": [0, 2],
      "paid_media_spend_e_alphas": [0.5, 3],
      "paid_media_spend_e_gammas": [0.3, 1],
      "paid_media_spend_e_scales": [0.0001, 0.1],
      "paid_media_spend_e_shapes": [0, 2],
      "paid_media_spend_f_alphas": [0.5, 3],
      "paid_media_spend_f_gammas": [0.3, 1],
      "paid_media_spend_f_scales": [0.0001, 0.1],
      "paid_media_spend_f_shapes": [0, 2],
      "paid_media_spend_g_alphas": [0.5, 3],
      "paid_media_spend_g_gammas": [0.3, 1],
      "paid_media_spend_g_scales": [0.0001, 0.1],
      "paid_media_spend_g_shapes": [0, 2],
      "paid_media_spend_h_alphas": [0.5, 3],
      "paid_media_spend_h_gammas": [0.3, 1],
      "paid_media_spend_h_scales": [0.0001, 0.1],
      "paid_media_spend_h_shapes": [0, 2],
      "paid_media_spend_j_alphas": [0.5, 3],
      "paid_media_spend_j_gammas": [0.3, 1],
      "paid_media_spend_j_scales": [0.0001, 0.1],
      "paid_media_spend_j_shapes": [0, 2],
      "paid_media_spend_i_alphas": [0.5, 3],
      "paid_media_spend_i_gammas": [0.3, 1],
      "paid_media_spend_i_scales": [0.0001, 0.1],
      "paid_media_spend_i_shapes": [0, 2],
      "paid_media_spend_k_alphas": [0.5, 3],
      "paid_media_spend_k_gammas": [0.3, 1],
      "paid_media_spend_k_scales": [0.0001, 0.1],
      "paid_media_spend_k_shapes": [0, 2],
      "train_size": [0.5, 0.8]
    },
    "calibration_input": {},
    "custom_params": [],
    "version": ["Robyn (dev) v3.10.4.9008 [R-4.3.0]"]
  },
  "ExportedModel": {
    "select_model": ["5_44_6"],
    "ts_validation": [false],
    "run_time": ["10.75 min"],
    "outputs_time": ["1.71 min"],
    "total_time": ["12.46 min"],
    "conv_msg": ["DECOMP.RSSD NOT converged", "NRMSE NOT converged"],
    "summary": [
      {
        "variable": "(Intercept)",
        "coef": 0,
        "decompPer": 0,
        "decompAgg": 0
      },
      {
        "variable": "trend",
        "coef": 0.82248556807,
        "decompPer": 0.8280661506,
        "decompAgg": 2896069343.56584
      },
      {
        "variable": "season",
        "coef": 0.96435146228,
        "decompPer": -0.0012488736315,
        "decompAgg": -4367796.74602932
      },
      {
        "variable": "weekday",
        "coef": 0.87577631075,
        "decompPer": -5.1374849601e-05,
        "decompAgg": -179677.827491431
      },
      {
        "variable": "holiday",
        "coef": 0.91971860576,
        "decompPer": -0.0052682273752,
        "decompAgg": -18425039.8173128
      },
      {
        "variable": "promotions_a",
        "coef": 2.7006268401,
        "decompPer": 0.014577753455,
        "decompAgg": 50984072.7684017
      },
      {
        "variable": "promotions_b",
        "coef": 4.2050466198,
        "decompPer": 0.0033324505301,
        "decompAgg": 11654875.4132919
      },
      {
        "variable": "promotions_c",
        "coef": 8.5600362421,
        "decompPer": 0.00011112369235,
        "decompAgg": 388642.765465745
      },
      {
        "variable": "subscriptions_a",
        "coef": 29.8122862751,
        "decompPer": 0.013916842977,
        "decompAgg": 48672611.815043
      },
      {
        "variable": "paid_media_spend_a",
        "coef": 0,
        "decompPer": 0,
        "decompAgg": 0,
        "performance": 0,
        "mean_response": 0,
        "mean_spend": 368.5836680054,
        "boot_mean": 0,
        "ci_low": 0,
        "ci_up": 0
      },
      {
        "variable": "paid_media_spend_b",
        "coef": 34684.4153165502,
        "decompPer": 0.0034637005592,
        "decompAgg": 12113907.8047741,
        "performance": 0.89208224987,
        "mean_response": 13428.4995707044,
        "mean_spend": 18178.5301204819,
        "boot_mean": 1.2033016095,
        "ci_low": 0.20915604745,
        "ci_up": 2.2091793329
      },
      {
        "variable": "paid_media_spend_c",
        "coef": 160210.920575209,
        "decompPer": 0.0052625339487,
        "decompAgg": 18405127.6909241,
        "performance": 139.9065600247,
        "mean_response": 882.4623966557,
        "mean_spend": 176.1084337349,
        "boot_mean": 126.9580996561,
        "ci_low": 85.1079787278,
        "ci_up": 168.3200972695
      },
      {
        "variable": "paid_media_spend_d",
        "coef": 255748.859828737,
        "decompPer": 0.013374037008,
        "decompAgg": 46774208.2560722,
        "performance": 5.561299248,
        "mean_response": 60531.0573175548,
        "mean_spend": 11259.2530120482,
        "boot_mean": 4.0048282221,
        "ci_low": 0,
        "ci_up": 8.9767514492
      },
      {
        "variable": "paid_media_spend_e",
        "coef": 17141.6536164307,
        "decompPer": 0.00026856983031,
        "decompAgg": 939293.13690535,
        "performance": 2.0215503119,
        "mean_response": 0.81066956727,
        "mean_spend": 622.0080321285,
        "boot_mean": 0.6987825578,
        "ci_low": 0,
        "ci_up": 4.924174236
      },
      {
        "variable": "paid_media_spend_f",
        "coef": 313406.012833488,
        "decompPer": 0.023740742019,
        "decompAgg": 83030607.039644,
        "performance": 13.8120075322,
        "mean_response": 123090.053626331,
        "mean_spend": 8047.4966532798,
        "boot_mean": 22.29415848,
        "ci_low": 0,
        "ci_up": 49.7089269385
      },
      {
        "variable": "paid_media_spend_g",
        "coef": 0,
        "decompPer": 0,
        "decompAgg": 0,
        "performance": 0,
        "mean_response": 0,
        "mean_spend": 2113.9236947791,
        "boot_mean": 0,
        "ci_low": 0,
        "ci_up": 0
      },
      {
        "variable": "paid_media_spend_h",
        "coef": 493290.052433987,
        "decompPer": 0.042181630708,
        "decompAgg": 147525566.002729,
        "performance": 4.7299865667,
        "mean_response": 262554.710016023,
        "mean_spend": 41752.9129852744,
        "boot_mean": 5.2533212822,
        "ci_low": 3.5816036696,
        "ci_up": 6.944767254
      },
      {
        "variable": "paid_media_spend_i",
        "coef": 0,
        "decompPer": 0,
        "decompAgg": 0,
        "performance": 0,
        "mean_response": 0,
        "mean_spend": 5366.8393574297,
        "boot_mean": 0,
        "ci_low": 0,
        "ci_up": 0
      },
      {
        "variable": "paid_media_spend_j",
        "coef": 590158.13061405,
        "decompPer": 0.058272940524,
        "decompAgg": 203803133.948303,
        "performance": 5.1332904344,
        "mean_response": 236674.451158445,
        "mean_spend": 53148.9170013387,
        "boot_mean": 4.4742345119,
        "ci_low": 2.3441302934,
        "ci_up": 6.5794940344
      },
      {
        "variable": "paid_media_spend_k",
        "coef": 0,
        "decompPer": 0,
        "decompAgg": 0,
        "performance": 0,
        "mean_response": 0,
        "mean_spend": 4574.0334672021,
        "boot_mean": 0,
        "ci_low": 0,
        "ci_up": 0
      }
    ],
    "errors": [
      {
        "rsq_train": 0.65900160112,
        "nrmse_train": 0.054262478985,
        "nrmse": 0.054262478985,
        "decomp.rssd": 0.22265580257,
        "mape": 0
      }
    ],
    "hyper_values": {
      "lambda": [86324.8902967888],
      "paid_media_spend_a_alphas": [2.508076623],
      "paid_media_spend_a_gammas": [0.63332630289],
      "paid_media_spend_a_scales": [0.045013384737],
      "paid_media_spend_a_shapes": [1.1837844198],
      "paid_media_spend_b_alphas": [2.057585828],
      "paid_media_spend_b_gammas": [0.80516993968],
      "paid_media_spend_b_scales": [0.065821954148],
      "paid_media_spend_b_shapes": [0.9355082856],
      "paid_media_spend_c_alphas": [2.897264612],
      "paid_media_spend_c_gammas": [0.69687106193],
      "paid_media_spend_c_scales": [0.032555313169],
      "paid_media_spend_c_shapes": [0.873088992],
      "paid_media_spend_d_alphas": [1.672958028],
      "paid_media_spend_d_gammas": [0.63003369497],
      "paid_media_spend_d_scales": [0.069238738534],
      "paid_media_spend_d_shapes": [1.1204616652],
      "paid_media_spend_e_alphas": [1.853094674],
      "paid_media_spend_e_gammas": [0.5925706252],
      "paid_media_spend_e_scales": [0.082758182666],
      "paid_media_spend_e_shapes": [1.0757785012],
      "paid_media_spend_f_alphas": [0.89862984675],
      "paid_media_spend_f_gammas": [0.99369529523],
      "paid_media_spend_f_scales": [0.080554939455],
      "paid_media_spend_f_shapes": [0.9192846244],
      "paid_media_spend_g_alphas": [1.5622074615],
      "paid_media_spend_g_gammas": [0.70362543732],
      "paid_media_spend_g_scales": [0.040584638287],
      "paid_media_spend_g_shapes": [1.4914186078],
      "paid_media_spend_h_alphas": [2.3884523665],
      "paid_media_spend_h_gammas": [0.56650138431],
      "paid_media_spend_h_scales": [0.0011839237023],
      "paid_media_spend_h_shapes": [1.4241365776],
      "paid_media_spend_j_alphas": [1.8831342945],
      "paid_media_spend_j_gammas": [0.64622927038],
      "paid_media_spend_j_scales": [0.087922187652],
      "paid_media_spend_j_shapes": [0.9473128736],
      "paid_media_spend_i_alphas": [1.4670505397],
      "paid_media_spend_i_gammas": [0.62123683963],
      "paid_media_spend_i_scales": [0.045122170722],
      "paid_media_spend_i_shapes": [1.9096182712],
      "paid_media_spend_k_alphas": [0.68414829522],
      "paid_media_spend_k_gammas": [0.51470547294],
      "paid_media_spend_k_scales": [0.053697880048],
      "paid_media_spend_k_shapes": [1.9006466748],
      "train_size": [1]
    },
    "hyper_updated": {
      "paid_media_spend_a_alphas": [0.5, 3],
      "paid_media_spend_a_gammas": [0.3, 1],
      "paid_media_spend_a_scales": [0.0001, 0.1],
      "paid_media_spend_a_shapes": [0, 2],
      "paid_media_spend_b_alphas": [0.5, 3],
      "paid_media_spend_b_gammas": [0.3, 1],
      "paid_media_spend_b_scales": [0.0001, 0.1],
      "paid_media_spend_b_shapes": [0, 2],
      "paid_media_spend_c_alphas": [0.5, 3],
      "paid_media_spend_c_gammas": [0.3, 1],
      "paid_media_spend_c_scales": [0.0001, 0.1],
      "paid_media_spend_c_shapes": [0, 2],
      "paid_media_spend_d_alphas": [0.5, 3],
      "paid_media_spend_d_gammas": [0.3, 1],
      "paid_media_spend_d_scales": [0.0001, 0.1],
      "paid_media_spend_d_shapes": [0, 2],
      "paid_media_spend_e_alphas": [0.5, 3],
      "paid_media_spend_e_gammas": [0.3, 1],
      "paid_media_spend_e_scales": [0.0001, 0.1],
      "paid_media_spend_e_shapes": [0, 2],
      "paid_media_spend_f_alphas": [0.5, 3],
      "paid_media_spend_f_gammas": [0.3, 1],
      "paid_media_spend_f_scales": [0.0001, 0.1],
      "paid_media_spend_f_shapes": [0, 2],
      "paid_media_spend_g_alphas": [0.5, 3],
      "paid_media_spend_g_gammas": [0.3, 1],
      "paid_media_spend_g_scales": [0.0001, 0.1],
      "paid_media_spend_g_shapes": [0, 2],
      "paid_media_spend_h_alphas": [0.5, 3],
      "paid_media_spend_h_gammas": [0.3, 1],
      "paid_media_spend_h_scales": [0.0001, 0.1],
      "paid_media_spend_h_shapes": [0, 2],
      "paid_media_spend_j_alphas": [0.5, 3],
      "paid_media_spend_j_gammas": [0.3, 1],
      "paid_media_spend_j_scales": [0.0001, 0.1],
      "paid_media_spend_j_shapes": [0, 2],
      "paid_media_spend_i_alphas": [0.5, 3],
      "paid_media_spend_i_gammas": [0.3, 1],
      "paid_media_spend_i_scales": [0.0001, 0.1],
      "paid_media_spend_i_shapes": [0, 2],
      "paid_media_spend_k_alphas": [0.5, 3],
      "paid_media_spend_k_gammas": [0.3, 1],
      "paid_media_spend_k_scales": [0.0001, 0.1],
      "paid_media_spend_k_shapes": [0, 2],
      "lambda": [0, 1],
      "train_size": [1]
    },
    "calibration_constraint": [0.1],
    "cores": [7],
    "iterations": [500],
    "trials": [5],
    "intercept_sign": ["non_negative"],
    "nevergrad_algo": ["TwoPointsDE"],
    "add_penalty_factor": [false],
    "seed": [123],
    "pareto_fronts": [5],
    "hyper_fixed": [false],
    "plot_folder": ["./output/Robyn_202309202111_init/"]
  }
}
library(Robyn)

dir <- '{REPLACE_WITH_FILES_DIRECTORY}' # e.g. /Users/ddempsey/Downloads/solution/
json_filename <- '{REPLACE_WITH_JSON_FILENAME}' # e.g. RobynModel-5_44_6.json

print('Reading csv files...')
train_filename <- 'train.csv'
train_filepath <- paste(dir, train_filename, sep="")
train <- read.csv(train_filepath)
events_filename <- 'events.csv'
events_filepath <- paste(dir, events_filename, sep="")
events <- read.csv(events_filepath)

print('Reading json file...')
json_filepath <- paste(dir, json_filename, sep="")
json_data <- robyn_read(json_filepath)

print('Recreating model...')
RobynRecreated <- robyn_recreate(
  json_file = json_data,
  dt_input = train,
  dt_holidays = events,
  quiet = FALSE)
InputCollectX <- RobynRecreated$InputCollect
OutputCollectX <- RobynRecreated$OutputCollect

print('Coefficents of recreated model...')
print(OutputCollectX$xDecompAgg[, c('rn', 'coef')])
gufengzhou commented 1 year ago

Run script and compare the print coefficients from the recreated model to the coefficients from the JSON. For example, paid media spend b has a coefficient of 34684.4153165502 in the JSON but a coefficent of 72573 in the recreated model.

Hi thanks for sharing this, yes I can confirm that I also see a different coef of 77336 for paid media spend b with your json, but also from different from your number. Looking deeper, the coefs are already fixed after fitting glmnet() and I haven't found out why the difference.

When I tried to reproduce this difference with the simulated data, meaning building a new model, exporting the json and compare it with the recreate, everything looks good though. I've just merged the digit change commit into main. Would you mind re-building it using robyn_run etc with the latest version and compare them again?

ddempseyr commented 1 year ago

@gufengzhou Thanks for the update. I rebuilt our container pointed to the latest commit, created a new model, and recreated the model.

Overall, the results (e.g. coefficients) are significantly closer after the last commit but not one-to-one. After the commit, the percent difference between the coefficients in the JSON and recreated model are typically within 1 percentage point of each other. For example, one channel has a coefficient of 206466.458775187 in the JSON and 206228 in the recreated model and another channel has 657455.846006155 in the JSON and 657973 in the recreated model.

Is it possible that precision/digits still needs to be slightly increased?

gufengzhou commented 1 year ago

Can you check the hyper_values in json against OutputCollect$resultHypParam to see if the hyperparameters are having the same values/digits?

ddempseyr commented 1 year ago

@gufengzhou I ran a test and it appears the JSON file has a higher scale relative to the scale in OutputCollect$resultHypParam. The JSON file has a scale of 10 while OutputCollect$resultHypParam has a scale of 7.

For example, a channel's "shape" hyperparmeter was 1.4995834062 in the JSON while 1.4995834 in OutputCollect$resultHypParam.

I would expect the difference in scale to be the source of the issue and making them align would resolve it.

ddempseyr commented 1 year ago

@gufengzhou On a side note, do you know whether there were any changes to main since the v3.10.3 CRAN release that would impact the date_range in the robyn_allocator?

We were previously installing Robyn from CRAN (v3.10.3), and after pointing our install to the c8786d87e90f0d615ca18139170567066f457cb2 commit, robyn_allocator results in an error whenever dates (e.g. c("2023-01-01", "2023-01-30") ) are supplied to the date_range parameter, but successfully runs when the date_range parameter is not passed. See the error below for more information. 

5icw741mqf-algo-1-y4t29 | >>> Running budget allocator for model ID 1_124_1 ...
5icw741mqf-algo-1-y4t29 | Date Window: 2023-01-01:2023-01-30 (30 days)
5icw741mqf-algo-1-y4t29 | Error in charToDate(x) : 
5icw741mqf-algo-1-y4t29 |   character string is not in a standard unambiguous format
5icw741mqf-algo-1-y4t29 | Calls: source ... ==.Date -> as.Date -> as.Date.character -> charToDate
gufengzhou commented 1 year ago

We were previously installing Robyn from CRAN (v3.10.3), and after pointing our install to the c8786d8 commit, robyn_allocator results in an error whenever dates (e.g. c("2023-01-01", "2023-01-30") ) are supplied to the date_range parameter, but successfully runs when the date_range parameter is not passed. See the error below for more information.

The latest commit should've solved this. would you please test?

gufengzhou commented 1 year ago

@gufengzhou I ran a test and it appears the JSON file has a higher scale relative to the scale in OutputCollect$resultHypParam. The JSON file has a scale of 10 while OutputCollect$resultHypParam has a scale of 7.

For example, a channel's "shape" hyperparmeter was 1.4995834062 in the JSON while 1.4995834 in OutputCollect$resultHypParam.

I would expect the difference in scale to be the source of the issue and making them align would resolve it.

would you mind provide your files again? I can't reproduce the hyppar difference of shape/scale

ddempseyr commented 12 months ago

@gufengzhou Thanks for the speedy response. I'll work on getting this over to you today.

ddempseyr commented 12 months ago

@gufengzhou Good news. I believe the issue with robyn_allocator has been resolved. We updated Robyn to the latest commit on main and we are no longer seeing errors when passing a start and end date into the date_range parameter in the robyn_allocator.

ddempseyr commented 12 months ago

@gufengzhou In regards to the discrepancy with recreated models, we rebuilt our Docker image with the Robyn install pointed to the latest commit. See the files below and script that can be used to compare the results.

Based on my troubleshooting, the hyperparameter values actually appear to be one-to-one across the original trained model, json, and the recreated model, but there is still small discrepancy in the coefficients across the original model and recreated model. Therefore, I believe the discrepancy with hyperparameters has been resolved and potentially some other discrepancy between the original and recreated model is causing the slight difference in coefficients.

R Script Comparing Values

library(Robyn)

dir <- '{REPLACE_WITH_FILES_DIRECTORY}' # e.g. /Users/ddempsey/Downloads/solution/
json_filename <- '{REPLACE_WITH_JSON_FILENAME}' # e.g. RobynModel-1_96_1.json

print('Reading csv files...')
print('Reading files outputted during initial model training...')
pareto_hyperparameters_filename <- 'pareto_hyperparameters.csv'
pareto_hyperparameters_filepath <- paste(dir, pareto_hyperparameters_filename, sep="")
pareto_hyperparameters <- read.csv(pareto_hyperparameters_filepath)
pareto_aggregated_filename <- 'pareto_aggregated.csv'
pareto_aggregated_filepath <- paste(dir, pareto_aggregated_filename, sep="")
pareto_aggregated <- read.csv(pareto_aggregated_filepath)

print('Reading files required to recreated model...')
train_filename <- 'train.csv'
train_filepath <- paste(dir, train_filename, sep="")
train <- read.csv(train_filepath)
events_filename <- 'events.csv'
events_filepath <- paste(dir, events_filename, sep="")
events <- read.csv(events_filepath)
json_filepath <- paste(dir, json_filename, sep="")
json_data <- robyn_read(json_filepath)

print('Recreating model...')
RobynRecreated <- robyn_recreate(
  json_file = json_data,
  dt_input = train,
  dt_holidays = events,
  quiet = FALSE)
InputCollectX <- RobynRecreated$InputCollectx
OutputCollectX <- RobynRecreated$OutputCollect

print('Comparing hyperparameters from original model, recreated model, and json...')
print('hyperparameters from original model...')
print(pareto_hyperparameters[pareto_hyperparameters$solID == '1_96_1',]$paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_scales)
print('hyperparameters from recreated model...')
print(OutputCollectX$resultHypParam[OutputCollectX$resultHypParam$solID == '1_96_1',]$paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_scales)
print('hyperparameters from json model...')
print(json_data$ExportedModel$hyper_values$paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_scales)

print('Coefficents of original model to ...')
print(pareto_aggregated[pareto_aggregated$solID == '1_96_1', c('rn', 'coef')])
print(OutputCollectX$xDecompAgg[OutputCollectX$xDecompAgg$solID == '1_96_1', c('rn', 'coef')])

RobynModel-1_96_1.json

{
  "InputCollect": {
    "date_var": ["report_date"],
    "dayInterval": [1],
    "intervalType": ["day"],
    "dep_var": ["dependent__sales__none__none__none"],
    "dep_var_type": ["revenue"],
    "prophet_vars": ["trend", "season", "weekday", "holiday"],
    "prophet_signs": ["default", "default", "default", "default"],
    "prophet_country": ["US"],
    "context_vars": ["context__promotions__91d121ab5ddd4e6cb58c18c81965f372__none__none", "context__promotions__c3963f42782b4f2fb5afbda351f6aee4__none__none", "context__promotions__fab0ae9533034baabe9918a054bc0b07__none__none", "context__subscriptions__c3963f42782b4f2fb5afbda351f6aee4__none__none"],
    "context_signs": ["default", "default", "default", "default"],
    "paid_media_vars": ["paid_media__exposure__2513227cfff64829b9b21191b64be723__none__display..none", "paid_media__exposure__2513227cfff64829b9b21191b64be723__none__search..none", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none", "paid_media__exposure__d203f70a3fa447ec822fe5e885763062__none__shopping..none"],
    "paid_media_signs": ["positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive", "positive"],
    "paid_media_spends": ["paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none", "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none", "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none"],
    "mediaVarCount": [11],
    "exposure_vars": ["paid_media__exposure__2513227cfff64829b9b21191b64be723__none__display..none", "paid_media__exposure__2513227cfff64829b9b21191b64be723__none__search..none", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown", "paid_media__exposure__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none", "paid_media__exposure__d203f70a3fa447ec822fe5e885763062__none__shopping..none"],
    "all_media": ["paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none", "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none", "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none"],
    "all_ind_vars": ["trend", "season", "weekday", "holiday", "context__promotions__91d121ab5ddd4e6cb58c18c81965f372__none__none", "context__promotions__c3963f42782b4f2fb5afbda351f6aee4__none__none", "context__promotions__fab0ae9533034baabe9918a054bc0b07__none__none", "context__subscriptions__c3963f42782b4f2fb5afbda351f6aee4__none__none", "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none", "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown", "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none", "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none"],
    "unused_vars": [],
    "window_start": ["2021-09-03"],
    "rollingWindowStartWhich": [1],
    "window_end": ["2023-10-04"],
    "rollingWindowEndWhich": [762],
    "rollingWindowLength": [762],
    "refreshAddedStart": ["2021-09-03"],
    "adstock": ["weibull_pdf"],
    "hyperparameters": {
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_alphas": [0.5, 3],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_gammas": [0.3, 1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_scales": [0.0001, 0.1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_shapes": [0, 2],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_alphas": [0.5, 3],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_gammas": [0.3, 1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_scales": [0.0001, 0.1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_shapes": [0, 2],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_alphas": [0.5, 3],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_gammas": [0.3, 1],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_scales": [0.0001, 0.1],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_shapes": [0, 2],
      "train_size": [0.5, 0.8]
    },
    "calibration_input": {},
    "custom_params": [],
    "version": ["Robyn (dev) v3.10.4.9011 [R-4.3.0]"]
  },
  "ExportedModel": {
    "select_model": ["1_96_1"],
    "ts_validation": [false],
    "run_time": ["15 min"],
    "outputs_time": ["3.73 min"],
    "total_time": ["18.73 min"],
    "conv_msg": ["DECOMP.RSSD NOT converged", "NRMSE NOT converged"],
    "summary": [
      {
        "variable": "(Intercept)",
        "coef": 3754998.5807831,
        "decompPer": 0.80529693786,
        "decompAgg": 2861308918.55672
      },
      {
        "variable": "trend",
        "coef": 0.095843430057,
        "decompPer": 0.096633946022,
        "decompAgg": 343351077.830573
      },
      {
        "variable": "season",
        "coef": 0.09955054392,
        "decompPer": -0.00026220751477,
        "decompAgg": -931652.245604457
      },
      {
        "variable": "weekday",
        "coef": 0.10881925938,
        "decompPer": -2.4004858534e-06,
        "decompAgg": -8529.1912316888
      },
      {
        "variable": "holiday",
        "coef": 0.075621443598,
        "decompPer": -0.00041922213242,
        "decompAgg": -1489542.51524325
      },
      {
        "variable": "context__promotions__91d121ab5ddd4e6cb58c18c81965f372__none__none",
        "coef": 0.57085358446,
        "decompPer": 0.0030330947498,
        "decompAgg": 10776920.4755261
      },
      {
        "variable": "context__promotions__c3963f42782b4f2fb5afbda351f6aee4__none__none",
        "coef": -1.9650272993,
        "decompPer": -0.0016085043365,
        "decompAgg": -5715193.47368502
      },
      {
        "variable": "context__promotions__fab0ae9533034baabe9918a054bc0b07__none__none",
        "coef": 1.2345070373,
        "decompPer": 1.5774654408e-05,
        "decompAgg": 56049.0885081595
      },
      {
        "variable": "context__subscriptions__c3963f42782b4f2fb5afbda351f6aee4__none__none",
        "coef": -1.5473073066,
        "decompPer": -0.00071425240034,
        "decompAgg": -2537817.62617163
      },
      {
        "variable": "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none",
        "coef": 76714.0765689108,
        "decompPer": 0.005939300012,
        "decompAgg": 21102988.5939494,
        "performance": 76.6456081892,
        "mean_response": 4047.3983359867,
        "mean_spend": 361.3280839895
      },
      {
        "variable": "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none",
        "coef": 243648.530145765,
        "decompPer": 0.028453976888,
        "decompAgg": 101100120.976162,
        "performance": 7.4451304101,
        "mean_response": 90240.1815350754,
        "mean_spend": 17820.6850393701
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded",
        "coef": 63179.4888822553,
        "decompPer": 0.0049122896349,
        "decompAgg": 17453907.3504527,
        "performance": 132.5901893864,
        "mean_response": 8365.2239310479,
        "mean_spend": 172.7532808399
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category",
        "coef": 31568.9782185146,
        "decompPer": 0.0014311390541,
        "decompAgg": 5084995.04566974,
        "performance": 0.6023867862,
        "mean_response": 1441.7197504189,
        "mean_spend": 11077.968503937
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none",
        "coef": 68170.8290036688,
        "decompPer": 0.0007078070325,
        "decompAgg": 2514916.52270988,
        "performance": 5.4126130396,
        "mean_response": 2511.900688857,
        "mean_spend": 609.7637795276
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto",
        "coef": 120934.438786772,
        "decompPer": 0.010141493913,
        "decompAgg": 36033847.4125978,
        "performance": 5.9570613997,
        "mean_response": 44728.7849943691,
        "mean_spend": 7938.2283464567
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded",
        "coef": 58186.2010820318,
        "decompPer": 0.004049016285,
        "decompAgg": 14386601.8395104,
        "performance": 9.0247866,
        "mean_response": 16743.9141396804,
        "mean_spend": 2092.0223097113
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category",
        "coef": 54608.6037779167,
        "decompPer": 0.0058004066553,
        "decompAgg": 20609485.1649304,
        "performance": 0.65481274016,
        "mean_response": 21172.6593215306,
        "mean_spend": 41304.2795275591
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown",
        "coef": 0,
        "decompPer": 0,
        "decompAgg": 0,
        "performance": 0,
        "mean_response": 0,
        "mean_spend": 5312.2034120735
      },
      {
        "variable": "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none",
        "coef": 338349.008544917,
        "decompPer": 0.033104530702,
        "decompAgg": 117624052.058077,
        "performance": 2.9441835632,
        "mean_response": 130928.821791957,
        "mean_spend": 52429.5682414698
      },
      {
        "variable": "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none",
        "coef": 30884.3831352287,
        "decompPer": 0.0034868734066,
        "decompAgg": 12389246.136548,
        "performance": 3.6259761352,
        "mean_response": 7492.8533391035,
        "mean_spend": 4483.9934383202
      }
    ],
    "errors": [
      {
        "rsq_train": 0.15608458929,
        "nrmse_train": 0.085399083407,
        "nrmse": 0.085399083407,
        "decomp.rssd": 0.33564515579,
        "mape": 0
      }
    ],
    "hyper_values": {
      "lambda": [7331504.86968052],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_alphas": [2.0586041565],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_gammas": [0.92519754717],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_scales": [0.020939103467],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_shapes": [1.0528318484],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_alphas": [1.7489948235],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_gammas": [0.68284700895],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_scales": [0.051637425965],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_shapes": [1.8148308202],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_alphas": [1.7325758937],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_gammas": [0.70396995107],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_scales": [0.06864500687],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_shapes": [0.8083000978],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_alphas": [1.7673814297],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_gammas": [0.46130178253],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_scales": [0.014784522229],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_shapes": [0.8539262908],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_alphas": [0.5779224732],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_gammas": [0.33475985501],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_scales": [0.025471538744],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_shapes": [1.1876021744],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_alphas": [1.364665793],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_gammas": [0.72391045522],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_scales": [0.057973930518],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_shapes": [1.381786837],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_alphas": [1.5846306728],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_gammas": [0.90154240027],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_scales": [0.074445351349],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_shapes": [0.010232815608],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_alphas": [0.5625106498],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_gammas": [0.48423539456],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_scales": [0.001524980589],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_shapes": [1.7105163954],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_alphas": [0.7515206675],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_gammas": [0.50798467109],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_scales": [0.036842691379],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_shapes": [0.0937410689],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_alphas": [1.8283520507],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_gammas": [0.66790983964],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_scales": [0.051346047925],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_shapes": [0.4666454488],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_alphas": [1.2221461678],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_gammas": [0.42514500257],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_scales": [0.07039658262],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_shapes": [1.4754943136],
      "train_size": [1]
    },
    "hyper_updated": {
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_alphas": [0.5, 3],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_gammas": [0.3, 1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_scales": [0.0001, 0.1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__display..none_shapes": [0, 2],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_alphas": [0.5, 3],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_gammas": [0.3, 1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_scales": [0.0001, 0.1],
      "paid_media__spend__2513227cfff64829b9b21191b64be723__none__search..none_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..branded_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..category_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_brands..none_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..auto_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..branded_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..category_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..none_shapes": [0, 2],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_alphas": [0.5, 3],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_gammas": [0.3, 1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_scales": [0.0001, 0.1],
      "paid_media__spend__4de0db3a3f0f48d988ca23d0313f0c1c__none__sponsored_products..unknown_shapes": [0, 2],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_alphas": [0.5, 3],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_gammas": [0.3, 1],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_scales": [0.0001, 0.1],
      "paid_media__spend__d203f70a3fa447ec822fe5e885763062__none__shopping..none_shapes": [0, 2],
      "lambda": [0, 1],
      "train_size": [1]
    },
    "calibration_constraint": [0.1],
    "cores": [3],
    "iterations": [500],
    "trials": [5],
    "intercept_sign": ["non_negative"],
    "nevergrad_algo": ["TwoPointsDE"],
    "add_penalty_factor": [false],
    "seed": [123],
    "pareto_fronts": [5],
    "hyper_fixed": [false],
    "plot_folder": ["./output/Robyn_202310051611_init/"]
  }
}

train.csv pareto_aggregated.csv pareto_hyperparameters.csv events.csv

ddempseyr commented 10 months ago

Hey @gufengzhou, any chance that you've looked into the remaining discrepancy in my pervious comment? Thanks in advance.

gufengzhou commented 10 months ago

Hi @ddempseyr , sorry for the delay. I've just built a new model using your data and recreated it, the coefs look identical. Not sure what caused your difference though. Can you test it again with the latest github version pls?

image

ddempseyr commented 9 months ago

@gufengzhou Can you provide the entire script used to generate that screenshot? The coefficients in your screenshot are different than the coefficients in the JSON. Is it possible that you retrained the model rather than referencing the JSON?

From my side, I'm still seeing the discrepancy after updating to "3.10.5.9007" and rerunning the script above.

image
ghltk commented 7 months ago

@ddempseyr @gufengzhou Hi there, I am encountering this issue as well. Is there any update here?

Thank you!

gufengzhou commented 5 months ago

Hi all, I've just built a new model using the latest dev version then recreated it and compared both direct outputs using both weibull_pdf as well as geometric. They look identical. I can imagine if you're using an "old json" it might cause some problems. Would you try? 

OutputCollectX$xDecompAgg %>% 
  select(rn, coef) %>% 
  rename(coef_refresh = "coef") %>% 
  left_join(
    OutputCollect$xDecompAgg %>% 
    filter(solID == select_model) %>% 
    select(rn, coef), by = "rn"
  )

OutputCollectX is recreated, while OutputCollect is "original".

image