search

pfmc-assessments / lingcod

2 stars 2 forks source link

try to speed up Hessian by adjusting memory allocation #66

Closed iantaylor-NOAA closed 3 years ago

iantaylor-NOAA commented 3 years ago

The ss.log files for models with Hessian show non-zero values for cmpdiff.tmp (as shown in the r4ss output at the bottom of this issue), indicating that the CMPDIF_BUFFER_SIZE is too small so stuff is getting written to the hard drive, slowing down the calculation. I'm not sure adequate additional memory could be added to make it go away, but it's worth running with a command-line argument like -cbs 700000000 (7e8) in case that helps.

SS has this default: gradient_structure::set_CMPDIF_BUFFER_SIZE(200000000); // 2e8 = about 0.2GB (from https://github.com/nmfs-stock-synthesis/stock-synthesis/blob/main/SS_global.tpl#L42)

More discussion about these settings is at: http://www.admb-project.org/documentation/faq/#memory

> mod.2021.s.007.001$logfile
      TempFile      Size
1 gradfil1.tmp         0
2 gradfil2.tmp         0
3 varssave.tmp         0
4  cmpdiff.tmp 400000016
> mod.2021.n.007.001$logfile
      TempFile      Size
1 gradfil1.tmp         0
2 gradfil2.tmp         0
3 varssave.tmp         0
4  cmpdiff.tmp 600000024
iantaylor-NOAA commented 3 years ago

Running north model with ss -cbs 1500000000 (1.5 billion or 15 with 8 zeros) resulted 0 values for all rows in the log file. It still took 51 minutes with the Hessian but would surely have taken longer otherwise. I think that's an allocation of ~1.5 GB (in addition to whatever other memory gets used by Stock Synthesis) but on our computers shouldn't be an issue even with multiple processes running.

I just added model_settings[["extras"]] <- "-cbs 1500000000" to run_investigatemodel.R in 1c3544e in the hope that this will speed up our profiles, retros, and jitters.

kellijohnson-NOAA commented 3 years ago

I just tried this for 2021.s.014.001 and it finished in 17 min and 7 seconds.

kellijohnson-NOAA commented 3 years ago

Added to run_investigatemodel without hessian in 7bbce41

kellijohnson-NOAA commented 3 years ago

@iantaylor-NOAA can we close this issue?