search

probcomp / hierarchical-irm

Probabilistic structure discovery for rich relational systems
Apache License 2.0
4 stars 2 forks source link

Test the CRP using the Stochastic Rank Statistic (SRS) #121

Open emilyfertig opened 4 months ago

emilyfertig commented 4 months ago

See https://arxiv.org/pdf/1902.10142

alex-lew commented 3 months ago

I think it would be awesome to have SRS-based diagnostics for inference, but not sure how highly I would prioritize it.

SRS could help us compare different posterior inference methods in DPMMs and possibly the more complex models (IRM, HIRM, full model — all are doing inference on partitions of entities or relations or both). It may also be useful as part of an integration test to see that inference is working as expected. But I suspect it would be easier to use simpler metrics at first (e.g. just looking at how joint probability of the current state changes over time in MCMC). These simpler metrics could be sufficient for us to discover bugs, even if they are not a great way of telling whether the MCMC chain has actually mixed.