nestauk / dap_aria_mapping

Mapping technology innovation to support The Advanced Research and Innovation Agency (ARIA)
MIT License
1 stars 0 forks source link

Validation: Sensitivity analysis with respect to model stability #46

Closed ampudia19 closed 1 year ago

ampudia19 commented 1 year ago

Image

Evaluate the ability of a method to yield stable topic distributions as the number of sample observations increases. A possible approach consists of:

  1. Identify a number of relevant entities that MUST be clustered together.
  2. On increasing sample sizes run approaches with unique hyperparam configurations, evaluate: (i) frequency with which elements in 1. are bundled together, (ii) reductions in noise following from #43, (iii) distribution of topic / sub-topic sizes.

NB. This corresponds to bullet-point Measure stability with different sample sizes: distribution of in Emily's comment on #12.