Weighted diagnostics - Githubissues

stan-dev / posterior

The posterior R package

https://mc-stan.org/posterior/

Other

167 stars 24 forks source link

Weighted diagnostics #357

Closed n-kall closed 1 month ago

n-kall commented 8 months ago

Summary

Adds weighted ESS, MCSE and Pareto diagnostics.

Copyright and Licensing

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

github-actions[bot] commented 8 months ago

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 77cdf1ace4251467cdfc990954363395878f5c0f is merged into rvar_weights:

:rocket:as_draws_array: 141ms -> 139ms [-2.13%, -0.83%]
:ballot_box_with_check:as_draws_df: 64.8ms -> 65.1ms [-1.67%, +2.43%]
:ballot_box_with_check:as_draws_list: 158ms -> 158ms [-0.74%, +1.11%]
:ballot_box_with_check:as_draws_matrix: 62.7ms -> 63.4ms [-0.79%, +2.97%]
:ballot_box_with_check:as_draws_rvars: 80.5ms -> 81.6ms [-0.13%, +2.77%]
:ballot_box_with_check:summarise_draws_100_variables: 709ms -> 710ms [-0.24%, +0.47%]
:ballot_box_with_check:summarise_draws_10_variables: 111ms -> 111ms [-1.44%, +0.6%] Further explanation regarding interpretation and methodology can be found in the documentation.

paul-buerkner commented 8 months ago

Thank you! @avehtari would you have time to review this PR?

n-kall commented 8 months ago

I should still add some documentation, but I think the core functionality is there (plus tests)

github-actions[bot] commented 8 months ago

This is how benchmark results would change (along with a 95% confidence interval in relative change) if d56a5c9442d5a437d3dbf2bebfbec51b06ef2f9e is merged into rvar_weights:

:rocket:as_draws_array: 151ms -> 149ms [-1.92%, -0.49%]
:exclamation::snail:as_draws_df: 72.8ms -> 73.7ms [+0.27%, +2.25%]
:ballot_box_with_check:as_draws_list: 168ms -> 168ms [-0.58%, +0.7%]
:ballot_box_with_check:as_draws_matrix: 67.8ms -> 67.3ms [-1.9%, +0.47%]
:ballot_box_with_check:as_draws_rvars: 84.5ms -> 84.2ms [-1.43%, +0.69%]
:ballot_box_with_check:summarise_draws_100_variables: 719ms -> 721ms [-0.14%, +0.53%]
:ballot_box_with_check:summarise_draws_10_variables: 116ms -> 116ms [-0.98%, +0.35%] Further explanation regarding interpretation and methodology can be found in the documentation.

github-actions[bot] commented 7 months ago

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 1079cefe5c83d6c9578d1f126a5887a1505aa344 is merged into rvar_weights:

:heavy_check_mark:as_draws_array: 137ms -> 137ms [-0.89%, +0.54%]
:heavy_check_mark:as_draws_df: 60.9ms -> 61.2ms [-1.14%, +2.31%]
:exclamation::snail:as_draws_list: 156ms -> 157ms [+0.53%, +1.79%]
:heavy_check_mark:as_draws_matrix: 60ms -> 59.4ms [-2.16%, +0.01%]
:heavy_check_mark:as_draws_rvars: 82ms -> 82.5ms [-0.5%, +1.7%]
:exclamation::snail:summarise_draws_100_variables: 707ms -> 741ms [+4.44%, +5.01%]
:heavy_check_mark:summarise_draws_10_variables: 110ms -> 111ms [-0.2%, +1.03%] Further explanation regarding interpretation and methodology can be found in the documentation.

n-kall commented 1 month ago

I think this can be merged into the weighted rvars branch, as @avehtari and I went through and reviewed this some time ago

paul-buerkner commented 1 month ago

Thanks. Merging now.