search

pbs-assess / sdmTMB

:earth_americas: An R package for spatial and spatiotemporal GLMMs with TMB
https://pbs-assess.github.io/sdmTMB/
183 stars 26 forks source link

Newton optimizer failing on MacOS #244

Closed nafreymueller closed 1 year ago

nafreymueller commented 1 year ago

Hi,

I am trying to run a spatiotemporal model of commercial whaling and am having issues with the model running on a Mac. I'm eventually going to scale this up with economic feedbacks, but am starting with environmental data and will add complexity later.

Here's the data table and code I'm using in the model

# data frame used for the model - there were no issues making the mesh objects
df_tot
     halfdecade cellID     x    y nstrike vessels SST SSS  ICE Bathy stock ocean UID time
  1:       1845    837  -350 2950       1      12 6.7  32 0.05   -66     3     1   1    1
  2:       1845    838  -250 2950       3      12 6.9  32 0.06   -85     3     1   2    1
  3:       1845    839  -150 2950       2      12 7.1  32 0.08   -96     3     1   3    1
  4:       1845    916  -450 2850       1      12 6.0  32 0.06   -25     3     1   4    1
  5:       1845    917  -350 2850       1      12 6.3  32 0.06     0     3     1   5    1
 ---                                                                                     
304:       1900   2263 -1750 1150       3      15 3.5  28 0.15  -267     3     1 217   11
305:       1900   1636  -450 1950       1      15 2.0  30 0.36   -47     3     1 218   11
306:       1900   1637  -350 1950       1      15 1.7  30 0.45   -54     3     1 219   11
307:       1900   1639  -150 1950       1      15 1.0  30 0.52   -69     3     1 220   11
308:       1905    773  1250 3050       2      16 8.0  33 0.05  -245     4     1 336   12

pac.model1 <- sdmTMB(
  formula = nstrike ~ 0 + ICE + SST, # formula. No intercept
  data = df_tot, # data for the model
  family = poisson(link = "log"), # could also try nbinom2(link = "log")
  spatial_varying = ~ 1 + ICE + SST, # allow intercepts and effect of ICE and SST to vary in space
  # time_varying = ~ 0 + steam,
  time = "time", # time variable
  spatial = "off", # whether spatial factors are on
  spatiotemporal = "ar1", # pointless having random spatial and random spatiotemporal features both on
  mesh = pac.mesh, # the mesh (with barrier)
  silent = FALSE, # shows the progress
  control = sdmTMBcontrol(newton_loops = 2L)
)

When I am running this on MacOS with an M1 chip, it fails quickly and returns the following error related to the Newton optimizer:

outer mgc:  8948 
outer mgc:  6586.897 
outer mgc:  1415.072 
outer mgc:  936.4172 
outer mgc:  99.16083 
outer mgc:  389.2509 
outer mgc:  262.0326 
outer mgc:  16.76071 
outer mgc:  2.537779 
outer mgc:  0.3803737 
outer mgc:  0.01172721 
outer mgc:  8.136504e-05 
iter: 1  value: -3583.513 mgc: 136.3152 ustep: 0.9999 
iter: 2  value: -3583.513 mgc: 0.01210122 ustep: 0.99995 
iter: 3  Error in newton(par = c(zeta_s = 0, zeta_s = 0, zeta_s = 0, zeta_s = 0,  : 
  Newton failed to find minimum.
iter: 1  value: -3583.513 mgc: 136.3152 ustep: 0.9999 
iter: 2  value: -3583.513 mgc: 0.01210122 ustep: 0.99995 
iter: 3  Error in newton(par = c(zeta_s = 0, zeta_s = 0, zeta_s = 0, zeta_s = 0,  : 
  Newton failed to find minimum.
In addition: Warning message:
In stats::nlminb(start = tmb_obj$par, objective = tmb_obj$fn, gradient = tmb_obj$gr,  :
  NA/NaN function evaluation
Error in ff(x, order = 1) : 
  inner newton optimization failed during gradient calculation
outer mgc:  NaN 
Error in stats::nlminb(start = tmb_obj$par, objective = tmb_obj$fn, gradient = tmb_obj$gr,  : 
  NA/NaN gradient evaluation

However, when running on a Linux server, the same code runs fine.

This may perhaps be an issue with running glmmTMB under the surface. I submitted a request there (https://github.com/glmmTMB/glmmTMB/issues/927)

I'm attaching the sessionInfo for both computers. Is there any obvious fix anyone can see?

# RStudio server on Linux
sessionInfo()
R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
 [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: Australia/Adelaide
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] INLAspacetime_0.1.6 sp_2.0-0            Matrix_1.4-0        ggplot2_3.4.2       spatialEco_2.0-1   
[6] sf_1.0-13           sdmTMB_0.3.0.9002   data.table_1.14.8  

loaded via a namespace (and not attached):
 [1] gtable_0.3.3        TMB_1.9.4           lattice_0.20-45     bspm_0.5.2          numDeriv_2016.8-1.1
 [6] vctrs_0.6.3         tools_4.3.0         generics_0.1.3      parallel_4.3.0      tibble_3.2.1       
[11] proxy_0.4-27        fansi_1.0.4         pkgconfig_2.0.3     KernSmooth_2.23-20  assertthat_0.2.1   
[16] lifecycle_1.0.3     compiler_4.3.0      farver_2.1.1        munsell_0.5.0       terra_1.7-37       
[21] codetools_0.2-18    class_7.3-20        pillar_1.9.0        nloptr_2.0.3        MASS_7.3-55        
[26] classInt_0.4-9      boot_1.3-28         nlme_3.1-155        tidyselect_1.2.0    mvtnorm_1.2-2      
[31] dplyr_1.1.2         splines_4.3.0       grid_4.3.0          colorspace_2.1-0    cli_3.6.1          
[36] magrittr_2.0.3      utf8_1.2.3          e1071_1.7-13        withr_2.5.0         scales_1.2.1       
[41] estimability_1.4.1  emmeans_1.8.7       lme4_1.1-34         coda_0.19-4         glmmTMB_1.1.7      
[46] rgdal_1.6-7         mgcv_1.8-39         visreg_2.7.0        rlang_1.1.1         Rcpp_1.0.10        
[51] xtable_1.8-4        glue_1.6.2          INLA_23.04.24       DBI_1.1.3           rstudioapi_0.14    
[56] minqa_1.2.5         R6_2.5.1            units_0.8-2

# Normal RStudio on MacOS 
sessionInfo()
R version 4.3.1 (2023-06-16)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.4.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Australia/Adelaide
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] INLA_23.06.29     sp_2.0-0          Matrix_1.6-0      ggplot2_3.4.2     spatialEco_2.0-1  sf_1.0-14         glmmTMB_1.1.7     sdmTMB_0.3.0      data.table_1.14.8

loaded via a namespace (and not attached):
 [1] utf8_1.2.3          generics_0.1.3      class_7.3-22        KernSmooth_2.23-22  lattice_0.21-8      lme4_1.1-34         magrittr_2.0.3      grid_4.3.1         
 [9] inlabru_2.8.0       e1071_1.7-13        DBI_1.1.3           mgcv_1.9-0          fansi_1.0.4         scales_1.2.1        codetools_0.2-19    numDeriv_2016.8-1.1
[17] cli_3.6.1           rlang_1.1.1         units_0.8-2         munsell_0.5.0       splines_4.3.1       withr_2.5.0         visreg_2.7.0        parallel_4.3.1     
[25] tools_4.3.1         nloptr_2.0.3        INLAspacetime_0.1.6 minqa_1.2.5         dplyr_1.1.2         colorspace_2.1-0    boot_1.3-28.1       assertthat_0.2.1   
[33] vctrs_0.6.3         R6_2.5.1            proxy_0.4-27        lifecycle_1.0.3     classInt_0.4-9      MASS_7.3-60         pkgconfig_2.0.3     terra_1.7-39       
[41] pillar_1.9.0        gtable_0.3.3        glue_1.6.2          Rcpp_1.0.11         tibble_3.2.1        tidyselect_1.2.0    rstudioapi_0.15.0   farver_2.1.1       
[49] nlme_3.1-162        TMB_1.9.5           compiler_4.3.1

Thanks for anyone who has insights.

Cheers, Nick

seananderson commented 1 year ago

Strange. It looks like it's successfully running the first stage of optimization on just the fixed effects but failing when it adds the random effects. Assuming you got that far, I doubt your issues installing glmmTMB are your issue (assuming you have a version installed). sdmTMB uses some utilities from glmmTMB (and is obviously inspired by it), but doesn't use it as part of the optimization.

Can you send me a minimal version of code and data I can experiment with? My email is in the DESCRIPTION file. I'm working mainly on a similar Mac setup.

I assume the basic examples in ?sdmTMB run for you?

nafreymueller commented 1 year ago

Thanks for getting to this quickly! I've gotten the basic examples to run without issue. Will send you an email with the code/data in a minute

nafreymueller commented 1 year ago

It looks like there's a solution! If you are going in between the arm and x86 versions of R on Apple Silicon using software like RSwitch (see this blog for details: https://www.dsquintana.blog/jags-apple-silicon-m1-mac/), this might be causing R to try to compile the x86 version for some packagers like so:

ld: warning: ignoring file /usr/local/clang8/lib/libc++.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: in '/usr/local/clang8/lib/libunwind.dylib', building for macOS-arm64 but attempting to link with file built for macOS-x86_64

If this applies, try re-installing the arm version of R for Apple Silicon.

In addition to this, I received a Mac Binary of sdmTMB from Sean that ran successfully - hopefully this may be available to others if they experience the same problem.

seananderson commented 1 year ago

Thanks for the update @nafreymueller! Glad you found a solution.