Gadget not printing param values or weights to Likelihood Output

[pfrater]

I'm not sure if this is an Rgadget or a Gadget issue, but when I run gadget.iterative without simulated annealing specified in the optimisation file Gadget sometimes doesn't print the 3rd section of the Likelihood Output for some components (as called by the -o flag; section 12.2 of the Gadget User Guide). It will print the output fully for most components, but not for all. This results in an error in lines 160-164 of View(gadget.iterative) (see below) because the data section in read.gadget.lik.out for those components is NULL, and gadget.iterative cannot commence with the final run.

res <- plyr::ldply(run.string, function(x) {
  tmp <- read.gadget.lik.out(paste(wgts, paste("lik", 
    paste(x, collapse = "."), sep = "."), sep = "/"))$data
  tmp[likelihood$weights$name[restr]]
})

I cannot seem to replicate the issue other than when simulated annealing is not specified in the optimisation file. When I call gadget from the command line and ask it to print the Likelihood output using the -o flag then it does so properly, which makes me think it's an Rgadget issue.

Any thoughts?

[bthe]

My guess would be that the optimization has not finished properly. How do the interim step parameter files, i.e. the params.NameOfComponent, look like? Are they empty?

[pfrater]

As far as I can tell this is not the case. The params files for the components that it happens to are all present. I am also able to run gadget -s -i WGTS/params.NameOfComponent -main WGTS/main.NameOfComponent -o WGTS/lik.nameOfComponent and everything prints out fine then.

[bthe]

I've tested this using the following optinfo settings:

[hooke]
hookeiter 40  ; number of hooke & jeeves iterations
hookeeps  1e-04 ; minimum epsilon, hooke & jeeves halt criteria
rho       0.5   ; value for the resizing multiplier
lambda    0     ; initial value for the step length

and gadget.iterative runs without a hitch. Would it be possible for you to share the model?

[pfrater]

Okay, I think I've figured it out. Turns out it was the parameters for those components; they were just decimal places above the upper bound which caused Gadget to spit an error. I must have accidentally called the wrong param files with the -i flag in my previous comment.

I'll just have to use run.final = FALSE followed by a function that resets the parameter values and then resume.final = TRUE to hack my way around it. Unless you want me to add a check for this into gadget.iterative?

[bthe]

hmm.. that shouldn't happen. I assume you have BFGS in your optinfofile, as it is impossible for H&J to do this. My solution this would simply be to remove BFGS from the optinfofile;)

[pfrater]

Just a quick note that I did also run into this issue when only using Hooke and Jeeves. I wonder if it has to do with using scientific notation instead of explicitly stating the full value.

The optinfofile:

[hooke]
hookeiter 50000  ; number of hooke & jeeves iterations
hookeeps  1e-04 ; minimum epsilon, hooke & jeeves halt criteria
rho       0.5   ; value for the resizing multiplier
lambda    0     ; initial value for the step length

One of the components parameter files that overstepped the upper boundary. Note the values for cod.bh.lambda:

; Gadget version 2.2.00-BETA running on pfrater Thu Oct  4 08:56:54 2018
; Hooke & Jeeves algorithm ran for 516 function evaluations
; and stopped when the likelihood value was 3131.2003
; because the convergence criteria were met
switch  value       lower   upper   optimise
 cod.linf       120.21716         100      150        1
cod.k           0.18        0.12     0.18        1
cod.bbin       3.0015259           1    1e+08        1
cod.bbin.mult      3.0005493           1    1e+08        1
cod.init.area.mult     10.006714           1      100        1
cod.init.age.mult      10.001831           1      100        1
cod.t0             0          -2        0        1
cod.mat.alpha    -0.16000251       -0.24    -0.16        1
cod.mat.l50     76.79513        51.2     76.8        1
cod.bh.mu   3.936379e+08     3.2e+08  4.8e+08        1
cod.bh.lambda   1.2803745e+08   8.536e+07 1.28e+08        1
cod.spr.alpha     0.20015259       1e-05        3        1
cod.spr.l50    39.397888           1       60        1
cod.aut.alpha     0.20015259       1e-05        3        1
cod.aut.l50    38.460388           1       60        1
 cod.comm.alpha   0.080050401        0.08     0.12        1
cod.comm.l50       55.757456          48       72        1

[pfrater]

Okay. I finally figured this out. The issue is specific to Gadget and has nothing to do with the optimization routines, but with how Gadget outputs the parameter files. In the output files Gadget will round to four significant figures, so any value x>=10000 gets put into scientific notation. Therefore, if a value has an upper bound of either 1000001 or 1000100 it will get rounded to 1e+06 in the output.

I had a parameter with an upper value of 128040000 and so Gadget was rounding to 1.28e+08. When gadget.iterative went to recall that model to get the LikelihoodOutput, then Gadget would spit an error (invisible to R) when the optimized value for that parameter was 128000000 < param < 128039999.

So, the moral of the story is...keep param values < 9999 or change to scientific notation and round to four digits.

[bthe]

Well, I would start thinking about using multipliers in the gadget formula as the optimizers don't like varying scales on parameters.