bad fit observed for specific data

[clarkliming]

recently I ran into a very bad fit on only one of my machines with BFGS optimizer. The data is uploaded, and the call is

mmrm::mmrm(
  chgus ~ bcva_bl + strata + trt * visit + csh(visit | id),
  data = data,
  optimizer = "BFGS"
)

the result looks like

but using L-BFGS-B everything looks much better

The session info can be found here

R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.6 (Ootpa)

Matrix products: default
BLAS/LAPACK: /apps/rocs/2020.08/cascadelake/software/OpenBLAS/0.3.9-GCC-9.3.0/lib/libopenblas_skylakexp-r0.3.9.so;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Amsterdam
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] vctrs_0.6.5      nlme_3.1-165     cli_3.6.1        rlang_1.1.1     
 [5] stringi_1.7.12   purrr_1.0.2      generics_0.1.3   assertthat_0.2.1
 [9] glue_1.6.2       bit_4.0.5        backports_1.4.1  TMB_1.9.13      
[13] fansi_1.0.4      grid_4.3.0       tibble_3.2.1     lifecycle_1.0.3 
[17] stringr_1.5.1    compiler_4.3.0   pkgconfig_2.0.3  Rcpp_1.0.10     
[21] lattice_0.21-8   mmrm_0.3.12      R6_2.5.1         utf8_1.2.3      
[25] tidyselect_1.2.0 pillar_1.9.0     Rdpack_2.4       parallel_4.3.0  
[29] rbibutils_2.2.13 magrittr_2.0.3   checkmate_2.2.0  Matrix_1.5-4    
[33] tools_4.3.0      bit64_4.0.5      arrow_16.1.0    

data.rds.gz

[danielinteractive]

Hm interesting, thanks @clarkliming . Is this looking any better with the latest TMB? And how is it with the default optimizer "L-BFGS-B"?

[clarkliming]

@danielinteractive yes the L-BFGS-B works well on this case, see the screenshot above. The reason to use BFGS is just following your script https://github.com/openpharma/mmrm/blob/8fb99ca3922b677ba699020faba094f9faa24124/simulations/missing-data-benchmarks/R/method/mmrm-wrapper.R#L50 (it says that it errors if using L-BFGS-B first)

[danielinteractive]

I see thanks @clarkliming , and does the new TMB change anything for this case or not?

[clarkliming]

I see thanks @clarkliming , and does the new TMB change anything for this case or not?

haven't had the chance to test the new TMB; currently ocean only supports R-4.3 however in rspm Matrix require at least R-4.4; installing previous version from source always fails while installing from binary also fails (tell me that not supported)

[danielinteractive]

Ah ok, maybe something to check with the Ocean team for help?

[danielinteractive]

@clarkliming At least here on a Windows laptop I get a good result with BFGS:

mmrm fit

Formula:     
chgus ~ bcva_bl + strata + trt * visit + csh(visit | id)
Data:        data (used 4187 observations from 600 
subjects with maximum 10 timepoints)
Covariance:  heterogeneous compound symmetry (11 variance parameters)
Inference:   REML
Deviance:    15772.96

Coefficients: 
 (Intercept)      bcva_bl      strata2      strata3 
 5.410293519 -0.001688518 -0.989200445  0.972129787 
         trt       visit2       visit3       visit4 
 0.529867484  0.385529338  0.492312456  0.767195714 
      visit5       visit6       visit7       visit8 
 1.000949167  1.255761630  1.644350117  1.756062232 
      visit9      visit10   trt:visit2   trt:visit3 
 1.985387524  2.286288313  0.270066643  0.924385656 
  trt:visit4   trt:visit5   trt:visit6   trt:visit7 
 1.382654349  1.868366516  2.300515152  2.815120004 
  trt:visit8   trt:visit9  trt:visit10 
 3.592565124  4.138062176  4.602394616 

Model Inference Optimization:
Converged with code 0 and message: No message provided.

With the latest mmrm package

R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 22631)

Matrix products: default

locale:
[1] LC_COLLATE=German_Switzerland.utf8 
[2] LC_CTYPE=German_Switzerland.utf8   
[3] LC_MONETARY=German_Switzerland.utf8
[4] LC_NUMERIC=C                       
[5] LC_TIME=German_Switzerland.utf8    

time zone: Asia/Taipei
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] mmrm_0.3.14

loaded via a namespace (and not attached):
 [1] assertthat_0.2.1  backports_1.5.0   R6_2.5.1         
 [4] tidyselect_1.2.1  bit_4.5.0         Matrix_1.7-0     
 [7] lattice_0.22-6    tzdb_0.4.0        magrittr_2.0.3   
[10] glue_1.7.0        bit64_4.5.2       generics_0.1.3   
[13] lifecycle_1.0.4   cli_3.6.3         Rdpack_2.6.1     
[16] TMB_1.9.15        arrow_17.0.0.1    vctrs_0.6.5      
[19] grid_4.4.1        compiler_4.4.1    rstudioapi_0.16.0
[22] purrr_1.0.2       rbibutils_2.2.16  tools_4.4.1      
[25] checkmate_2.3.2   nlme_3.1-164      Rcpp_1.0.13      
[28] rlang_1.1.4 

[clarkliming]

@danielinteractive yes this bad fit only occurs on one of my machine, has not got the time to investigate