Passing variable into Cost function

[nagydavid]

Dear Florian,

My question would be that how can I pass an extra variable into the likelihood function. I tried the "variable" setup in a createBayesianSetup but it through an error.

I have a fairly complicated cost function.

I would like to pass further the parent function (variables (RunFile, PathtoDaisy, ctrldaisy)), which can be called later for the Cost.optim.D.

The Cost.Optim.D can receive a vector containing the proposed parameter set and running an external model code, when it finishes, R reads the output, crunch the data and calculate the cost and return it. In my case it is NSE.

I used this setup (functions,variables) for DEoptim optimization.

`Bayes.f<-function(RunFile,showLogFile,PathToDaisy,ctrldaisy){

  closeAllConnections()
  set.seed(1)
  p.config <- ctrldaisy$p.config
  param_matrix<-fread(p.config)

  #DEoptim contol parameters

  Base.Functions<-list("CheckParameters","runDaisy","f.update","updateParameters","f.cost","tryCatch","daisy.PID","gof")

  My.Packages <- list("data.table", "hydroGOF", "lubridate","RDaisy","birk","processx","stringr","BayesianTools")

  My.Functions <- list("maxmin","read.optim.25.swct","read.optim.60.swct","read.optim.90.swct",
                    "read.optim.110.swct","read.optim.190.swct","read.optim.210.swct","read.optim.DS",
                    "read.optim.DS_Crop","read.optim.HV.y", "read.optim","Cost.optim_D", "Cost.optim_D.DS",
                    "DaisyMorris","write.table", "export_pop","export_pop_failed","str_remove","file.path")

  My.ParVar<- list("ctrldaisy")

  #DeOptim calibration

  lowR <- param_matrix[name %in% ctrldaisy$param_sens, ]$min
  uppR <- param_matrix[name %in% ctrldaisy$param_sens, ]$max
  dflt <- param_matrix[name %in% ctrldaisy$param_sens, ]$dflt

  prior <- createUniformPrior(lower = lowR, upper = uppR, best = dflt)

  opts <- list(packages = My.Packages, variables =  c(Base.Functions,My.Functions,My.ParVar), dlls = NULL)

  bayesianSetup <- createBayesianSetup(likelihood = Cost.optim_D,
                                       prior = prior,
                                       names = param_matrix[name %in% ctrldaisy$param_sens, ]$name,
                                       #parallel = "external",
                                       parallelOptions=opts
                                       )

  settingsBT <- list(iterations = 4, adapt = FALSE)

  runMCMC(bayesianSetup = bayesianSetup,sampler = "DREAMzs", settings = settingsBT)

  stopParallel(bayesianSetup)

}`

[woodwards]

You can get variables from other environments (e.g. .GlobalEnv) using the get function in R.

a <- 1
cost <- function(){
  a <- get("a", globalenv())
  print(a)
}
cost()
#> [1] 1

^{Created on 2020-05-08 by the reprex package (v0.3.0)}

[nagydavid]

Thank you, I know this method, however I would like to know how to pass variable properly to the paralleloptions=list(packages=“all”, variables=“all”)

[woodwards]

You can do it using clusterExport as I have in the following example

https://github.com/woodwards/basgra_nz/blob/master/scripts/BC_BASGRA_BT.R

[florianhartig]

Hi David,

I'm not 100% sure if I understand correctly what you are doing. You want to pass a model path or something like that to your likelihood?

Yes, what Simon says should still works, also in parallel, as long as you properly export the global variables.

A somewhat more elegant and safe solution would be to write a function (in CS, we call this a factory) that generates new likelihoods with the variable that you want included, and pass those on to the BayesianSetup. I haven't tested this, but it should be something like:

factory <- function(variable){

  likelihood <- function(pars){
    return(list(pars, variable))
  }
  return(likelihood)
}

likelihood = factory("path")

likelihood(3)

In this way, you can generate a new likelihood any time you want to change your variable. Read more on factories here https://adv-r.hadley.nz/function-factories.html

[nagydavid]

Hi Florin thank you for your answer.

What I would like to achieve.:

Cost.optim_D <- function(x = NULL,RunFile,showLogFile,PathToDaisy,ctrluser,ctrldaisy){

  if(is.null(x) == FALSE){
    ctrldaisy$p <- x
    ctrldaisy$ind <- Sys.getpid()}

  Current_param<-data.table(t(ctrldaisy$p))
  names(Current_param) <- str_remove(ctrldaisy$param_sens, "[$]")

  print(Current_param)

  IsTimeouT<-FALSE

  IsTimeouT<-f.cost(RunFile, showLogFile, PathToDaisy, ctrluser, ctrldaisy)

  pop_suceed <- read.optim(ctrluser, ctrldaisy)
 return(pop_suceed[obj=="sum"]$NSE_inv)

The cost function is my "likelihood" function:

• the "x" variable able to accept external parameter sets
• runs the external model inside the f.cost function.
• reading the outputs of the model and calculates the log-transformed Nash -Sutcliffe Efficiency based on Cheng et al. 2014 [(http://folk.uio.no/chongyux/papers_SCI/jhydrol_38.pdf] )
• The function requires RunFile, showLogFile, PathToDaisy, ctrluser, ctrldaisy, These variables are in the environment of Bayes.f, but I don't know how can I pass it to

  bayesianSetup <- createBayesianSetup(likelihood = Cost.optim_D,
                                       prior = prior,
                                       names = param_matrix[name %in% ctrldaisy$param_sens, ]$name,
                                       #parallel = "external",
                                       parallelOptions=opts
                                       )

I know it may be a trivial question or I am missing something.

[florianhartig]

Hi David, as I say, BT does not currently support passing on arguments to the likelihood (there are certain technical reasons why we don't want to support this, among them that the BayesianSetup would not be able to execute a likelihood "on its own" without a further input).

So, you have 3 options

• hardcode the paths in your likelihood (save but not flexible)
• create a global variable, as suggested by Simon (flexible and easy, but a bit dangerous)
• Create a factory, as suggested by me (a bit more complicated, but safer, values will also be saved if you save the setup etc.)