How to interpret the output?

[richelbilderbeek]

Thanks to the BEAST2 devs (I guess mostly @rbouckaert?) for this nice piece of work!

What I'd like to see documented, is how to exactly interpret the output.

For example, when running this zipped XML file, I get:

richel@lubuntu:~$ ./beast nested_sampling.xml 
[...]
Counting 19 parameters
replacing logger tracelog with NSLogger
Writing file anthus_aco_sub.trees
         Sample      posterior ESS(posterior)     likelihood          prior
Writing file anthus_aco_sub.log
Start likelihood 0: -225.2488738545589 
1 particles initialised
              0      -140.6759              N      -157.4874        16.8115 --
ML: -158.32601908408833 Information: 0.8385606384288167
              1      -124.4630         2.0         -143.0242        18.5611 --
ML: -144.86277038390745 Information: 1.8385386453449868
              2      -124.6624         3.0         -141.6496        16.9872 --
ML: -143.96492404000415 Information: 1.7551898946725117
[More ML estimates]
             25      -124.7614        15.4         -137.0127        12.2512 --
ML: -141.98047696086337 Information: 2.8349216687952605
             26      -124.7070        15.9         -137.0124        12.3054 --
ML: -141.98047696054587 Information: 2.8349216691561026
             27      -124.6071        16.2         -137.0106        12.4035 --
ML: -141.98047696042886 Information: 2.8349216692884056
28<=10000&& (28<2.0*2.8349216692884056*1||Math.abs(-141.98047696042886 - -141.98047696054587)/Math.abs(-141.98047696042886) > 1.0E-12
Finished in 28 steps!
Marginal likelihood: -141.31236790862764 (bootstrap SD=1.4240215445106443)
Marginal likelihood: -140.98848039195855 (subsample SD=1.8638865000383444)
Marginal likelihood: -141.1644574217485(1.1601428428211327)
Information: 2.8349216692884056
SD: 1.6837225630395305

Operator                                                    Tuning    #accept    #reject      Pr(m)  Pr(acc|m)
ScaleOperator(YuleBirthRateScaler.t:anthus_aco_sub)        0.75000       4797        998    0.04000    0.82778 Try setting scaleFactor to about 0.562
ScaleOperator(YuleModelTreeScaler.t:anthus_aco_sub)        0.50000       1116       4748    0.04000    0.19031 
ScaleOperator(YuleModelTreeRootScaler.t:anthus_aco_sub)    0.50000        887       4915    0.04000    0.15288 
Uniform(YuleModelUniformOperator.t:anthus_aco_sub)               -      11869      46191    0.40000    0.20443 
SubtreeSlide(YuleModelSubtreeSlide.t:anthus_aco_sub)       1.00000        128      28742    0.20000    0.00443 Try decreasing size to about 0.5
Exchange(YuleModelNarrow.t:anthus_aco_sub)                       -       3007      26046    0.20000    0.10350 
Exchange(YuleModelWide.t:anthus_aco_sub)                         -        180       5641    0.04000    0.03092 
WilsonBalding(YuleModelWilsonBalding.t:anthus_aco_sub)           -         92       5643    0.04000    0.01604 

     Tuning: The value of the operator's tuning parameter, or '-' if the operator can't be optimized.
    #accept: The total number of times a proposal by this operator has been accepted.
    #reject: The total number of times a proposal by this operator has been rejected.
      Pr(m): The probability this operator is chosen in a step of the MCMC (i.e. the normalized weight).
  Pr(acc|m): The acceptance probability (#accept as a fraction of the total proposals for this operator).

Total calculation time: 0.823 seconds
End likelihood: 12.388056869575024
Producing posterior samples

Marginal likelihood: -141.3000915386624 sqrt(H/N)=(NaN)=?=SD=(1.4593051179523482) Information: -0.19242514244311712
Max ESS: 5.659472090348357

Processing 28 trees from file.
Log file written to anthus_aco_sub.posterior.trees
Done!

Marginal likelihood: -141.31271098696405 sqrt(H/N)=(NaN)=?=SD=(1.462246661656647) Information: -0.1353639346412825
Max ESS: 5.4945059635784155

Log file written to anthus_aco_sub.posterior.log
Done!

Which of these lines is the marginal likelihood I should use (all copied from above)?

1. -141.31236790862764, from Marginal likelihood: -141.31236790862764 (bootstrap SD=1.4240215445106443)
2. -140.98848039195855, from Marginal likelihood: -140.98848039195855 (subsample SD=1.8638865000383444)
3. -141.1644574217485, from Marginal likelihood: -141.1644574217485(1.1601428428211327)
4. -141.3000915386624, from Marginal likelihood: -141.3000915386624 sqrt(H/N)=(NaN)=?=SD=(1.4593051179523482) Information: -0.19242514244311712
5. -141.31271098696405, from Marginal likelihood: -141.31271098696405 sqrt(H/N)=(NaN)=?=SD=(1.462246661656647) Information: -0.1353639346412825

Which of these lines is the marginal likelihood error estimation to use?

1. 1.4240215445106443, from Marginal likelihood: -141.31236790862764 (bootstrap SD=1.4240215445106443)
2. 1.8638865000383444, from Marginal likelihood: -140.98848039195855 (subsample SD=1.8638865000383444)
3. 1.6837225630395305, from SD: 1.6837225630395305
4. 1.4593051179523482, from Marginal likelihood: -141.3000915386624 sqrt(H/N)=(NaN)=?=SD=(1.4593051179523482) Information: -0.19242514244311712
5. 1.462246661656647, from Marginal likelihood: -141.31271098696405 sqrt(H/N)=(NaN)=?=SD=(1.462246661656647) Information: -0.1353639346412825

Which of these lines is the Effective Sample Size to use?

1. 5.659472090348357, from Max ESS: 5.659472090348357
2. 5.4945059635784155, from Max ESS: 5.4945059635784155

Thanks, @richelbilderbeek

[rbouckaert]

Hi Richel, All estimates are stochastic, so they slightly differ every time they are re-estimated. So, from that perspective, any of the marginal likelihoods and SDs are valid, and you can take your favourite. The nice thing of nested sampling is that it comes with an estimate of the SD of the marginal likelihood, so that gives you a sense of how confident you can be in the ML estimate.

Likewise, the Max ESS values are stochastic estimates, and these numbers are used to generate a posterior sample of size equal to the Max ESS. But in practice, ESSs can be much smaller (depending on operator weights for example, just like with normal MCMC). The Max ESS value is just a theoretical maximum number. What the true ESS is is an open question.

Cheers, Remco

[richelbilderbeek]

Great, I can work with that! Thanks!