First draft of multiple chain discussion document

Puzzled-Face commented 5 months ago

First draft of the discussion of implementing multiple MCMC chains. This focusses on how to adapt the mcmc method, not on potential downstream consequences.

As I explain, I favour creation of (a) new ChainInitialiser class(es). As I wrote the document, I realised the current handling of some JAGS options (eg thin and n.iter) is sub-optimal. The fix is relatively straightforward. I will start work on it soon, but would appreciate feedback on the general approach I am proposing as I do so.

github-actions[bot] commented 5 months ago

Code Coverage Summary

Filename                          Stmts    Miss  Cover    Missing
------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
R/checkmate.R                        87       2  97.70%   72, 432
R/crmPack-package.R                   4       0  100.00%
R/CrmPackClass-methods.R              5       0  100.00%
R/Data-class.R                      156       5  96.79%   43, 565-566, 572-577
R/Data-methods.R                    267       0  100.00%
R/Data-validity.R                   144       1  99.31%   21
R/Design-class.R                    396       0  100.00%
R/Design-methods.R                 2766    1036  62.55%   563-567, 591-594, 602-621, 626-655, 659-660, 662, 677-685, 689, 701-720, 1122-1126, 1253, 1267-1271, 1333, 1523, 1740, 1767-1770, 1779-1790, 1794-1813, 1824-1828, 1834-1846, 2109, 2134-2137, 2144-2155, 2159-2178, 2190-2194, 2200-2212, 2496-2499, 2527-2530, 2538-2550, 2553-2564, 2568-2604, 2621-2630, 2636-2651, 2676, 2717-2718, 2981-3451, 3546-3557, 3560-3571, 3575-3611, 3628-3637, 3645-3660, 3698, 3740-3741, 3989-4545, 4686, 4774
R/Design-validity.R                  34      10  70.59%   47-56
R/fromQuantiles.R                   172      52  69.77%   294-362
R/helpers_broom.R                    74      10  86.49%   30, 34-35, 37-38, 40, 81, 102-104
R/helpers_covr.R                     23       0  100.00%
R/helpers_data.R                     96       1  98.96%   139
R/helpers_design.R                  126      42  66.67%   22, 77-129, 250, 255-259
R/helpers_jags.R                     77       0  100.00%
R/helpers_knitr_CohortSize.R        109       0  100.00%
R/helpers_knitr_Design.R            346       8  97.69%   24-29, 323-324
R/helpers_knitr_GeneralData.R       190      59  68.95%   75-78, 91-94, 107-110, 122-125, 138-141, 163-165, 248-252, 261-306, 354
R/helpers_knitr_GeneralModel.R      730       3  99.59%   105, 401, 813
R/helpers_knitr_Increments.R        165       1  99.39%   305
R/helpers_knitr_NextBest.R          312       0  100.00%
R/helpers_knitr_SafetyWindow.R      142       1  99.30%   103
R/helpers_knitr_Stopping.R          264       0  100.00%
R/helpers_knitr.R                    12       0  100.00%
R/helpers_model.R                    85       4  95.29%   38, 89-90, 139
R/helpers_rules.R                   428       0  100.00%
R/helpers_samples.R                   5       0  100.00%
R/helpers_simulations.R              27       0  100.00%
R/helpers.R                         214      61  71.50%   107-127, 162-178, 200-302, 337-351
R/logger.R                           11       0  100.00%
R/mcmc.R                            290      18  93.79%   92-97, 376-377, 387, 389-390, 393-396, 579-580, 669, 675, 733
R/McmcOptions-class.R                22       0  100.00%
R/McmcOptions-methods.R               8       1  87.50%   43
R/McmcOptions-validity.R             42       0  100.00%
R/Model-class.R                    1062     166  84.37%   145-147, 216-218, 222-224, 283-285, 357-359, 363-365, 444-446, 513-515, 577-581, 584-587, 690-693, 697-698, 813-817, 937-939, 943-951, 1096-1098, 1103-1106, 1110-1113, 1229-1233, 1235-1238, 1242-1245, 1248, 1409-1419, 1424-1430, 1585-1588, 1594-1601, 1758, 1767, 1776, 1785, 1794-1799, 1935, 1944, 1953, 1961-1963, 2807-2836, 2840-2846, 2853-2857, 2862, 2969-2982, 3008, 3104-3106, 3110, 3203-3205, 3209, 3278-3290, 3308, 3368-3370, 3372-3373, 3376-3381
R/Model-methods.R                   505      38  92.48%   78, 233-238, 809-854, 1175-1184
R/Model-validity.R                  443      16  96.39%   430-433, 442-445, 596-604
R/ModelParams-class.R                17       0  100.00%
R/ModelParams-validity.R             21       0  100.00%
R/Rules-class.R                     463       0  100.00%
R/Rules-methods.R                  1561     182  88.34%   889, 892, 895, 1010, 1013, 1016, 1136-1139, 1173, 1276-1279, 1314, 2582-2590, 2614-2621, 2784-2793, 3109-3118, 3251-3494, 3781, 3785
R/Rules-validity.R                  448      30  93.30%   684-723
R/Samples-class.R                     6       0  100.00%
R/Samples-methods.R                1188      21  98.23%   410-420, 648, 1665-1666, 1698, 1711, 1893, 2223-2228
R/Samples-validity.R                 10       0  100.00%
R/Simulations-class.R               208     103  50.48%   171-182, 419-493, 744-747, 943-976, 1004
R/Simulations-methods.R            1617    1473  8.91%    65-350, 406, 416-435, 448-453, 500-509, 674-2969
R/Simulations-validity.R             75      75  0.00%    20-168
R/utils.R                             6       0  100.00%
TOTAL                             15459    3419  77.88%

Diff against main

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: c4467ee4724e634c0c3c724e056065dcfb457e37

Minimum allowed coverage is 80%

:recycle: This comment has been updated with latest results

github-actions[bot] commented 5 months ago

Unit Tests Summary

1 files 45 suites 2m 24s :stopwatch: 1 346 tests 1 189 :white_check_mark: 157 :zzz: 0 :x: 6 560 runs 6 365 :white_check_mark: 195 :zzz: 0 :x:

Results for commit c4467ee4.

:recycle: This comment has been updated with latest results.

github-actions[bot] commented 5 months ago

Unit Test Performance Difference

Test Suite	$Status$	Time on `main`	$±Time$	$±Tests$	$±Skipped$	$±Failures$	$±Errors$
Design-methods	💔	$37.94$	$+2.00$	$0$	$0$	$0$	$0$
Rules-methods	💔	$29.75$	$+1.16$	$0$	$0$	$0$	$0$

Results for commit 0f3314429c02143dd6064cc920c033c6315169e2

♻️ This comment has been updated with latest results.

Puzzled-Face commented 5 months ago

Thanks @Puzzled-Face , very thorough document! Maybe I missed it when browsing through, but can you add a simple example how the new user interface would look like when they want to use multiple chains with default or custom starting values?

Is "thorough" a euphemism for "long"? ;-) It was considerably more work than I expected, but I think the effort was worthwhile.

There's an implicit example of using multiple chains with default values in the section on mcmc towards the end of the document [line 996 of the source document]:

threeChains <- mcmc(emptyData, myModel, options = LegacyChainOptions(3))

I'll add a more direct example immediately ~before~after "Other ChainOptions classes"...

openpharma / crmPack