Core dump when deallocating already freed memory from xpools

[johnoel]

xpools allocates a large chuck of memory to be used in ivector and dvector types. It is turned on by default by the macro USE_VECTOR_SHAPE_POOL and is a static type. ss.tpl has ivectors and dvectors in the GLOBALS_SECTION. When the C++ program completes the main function, it will call destructors for all global and static instances. Since ivectors and dvectors point to memory allocated by xpools, they should be deallocated before the static instance of xpools. This is not the case if ivectors and dvectors are global. For ss.tpl, the xpools destructor will be called first, then the ivectors and dvectors types. This will result in accessing memory that was already deallocated which caused the memory errors.

Issue reported by Kathryn

[johnoel]

Merged #139

[johnoel]

The core dump is caused by the ivector destructor that is accessing data from a deallocated memory pool. This occurs after leaving the main function. Previous commits to use flags to avoid the error still resulted in a core dump. The last approach was to disable vector shape pooling and just allocate the shape directly from memory. The results were no core dump. Below are the runtime results...

ss run output using macos with debug symbols

With vector shape pooling

In final section 
Finish time: Sun Jul 12 07:30:44 2020
Elapsed time: 0 hours, 20 minutes, 40 seconds.
Final gradient: 0.000816368

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11
./ss  1243.97s user 3.06s system 99% cpu 20:48.91 total

Without vector shape pooling

In final section 
Finish time: Sun Jul 12 08:00:51 2020
Elapsed time: 0 hours, 22 minutes, 7 seconds.
Final gradient: 0.000816368

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11
./ss  1331.73s user 2.92s system 99% cpu 22:16.10 total

Note: ss ran a bit faster with pooling which was expected.

ss run output using centos 7 with debug symbols

With vector shape pooling

In final section 
Finish time: Sun Jul 12 08:27:29 2020
Elapsed time: 0 hours, 7 minutes, 54 seconds.
Final gradient: 0.000190918

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11
Segmentation fault (core dumped)

real    7m59.212s
user    7m56.670s
sys 0m1.263s

Without vector shape pooling

In final section 
Finish time: Sun Jul 12 08:59:52 2020
Elapsed time: 0 hours, 8 minutes, 52 seconds.
Final gradient: 0.000190918

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11

real    8m57.524s
user    8m55.497s
sys 0m1.150s

[johnoel]

Time results for admb-12.0 with debug symbols

MacOS

With pooling

In final section 
Finish time: Mon Jul 13 08:54:41 2020
Elapsed time: 0 hours, 20 minutes, 30 seconds.
Final gradient: 0.000816368

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11
./ss  1234.24s user 3.42s system 99% cpu 20:39.56 total

Without pooling

In final section 
Finish time: Mon Jul 13 09:28:27 2020
Elapsed time: 0 hours, 22 minutes, 9 seconds.
Final gradient: 0.000816368

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11
./ss  1333.73s user 3.02s system 99% cpu 22:18.17 total

Redhat 7

With pooling

In final section 
Finish time: Mon Jul 13 10:53:04 2020
Elapsed time: 0 hours, 19 minutes, 27 seconds.
Final gradient: 0.000190918

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11

real    19m35.631s
user    19m33.058s
sys 0m1.203s

Without pooling

In final section 
Finish time: Mon Jul 13 11:26:44 2020
Elapsed time: 0 hours, 19 minutes, 56 seconds.
Final gradient: 0.000190918

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11

real    20m4.934s
user    20m2.210s
sys 0m1.221s

[johnoel]

Time results for admb-12.2pre2 with debug symbols

Redhat 7

Without pooling

In final section 
Finish time: Mon Jul 13 12:02:37 2020
Elapsed time: 0 hours, 20 minutes, 22 seconds.
Final gradient: 0.000190918

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11

real    20m31.770s
user    20m28.688s
sys 0m1.289s

With pooling

In final section 
Finish time: Mon Jul 13 12:46:04 2020
Elapsed time: 0 hours, 19 minutes, 56 seconds.
Final gradient: 0.000190918

 finished COVAR.SSO
call bmark
 finished forecast for reporting
 finished STD quantities for reporting
 write mcmc headers 
 finished posteriors reporting
 finished appending to cumreport.sso 
 finished SS_summary.sso
 finished SS_summary.sso 
 finished SIStable.sso 
 writing big output now 
 finished report.sso

!!  Run has completed  !!            See warning.sso for N warnings: 11
Segmentation fault (core dumped)

real    20m5.454s
user    20m2.107s
sys 0m1.332s