search

alan-turing-institute / ARC-SPICE

Sample Level, Pipeline Introduced Cumulative Errors (SPICE). Investigating methods for generalisable measurement of cumulative errors in multi-model (and multi-modal) ML pipelines.
MIT License
0 stars 0 forks source link

Topic Classification Component #13

Closed lannelin closed 1 week ago

lannelin commented 3 weeks ago

Topic classification

Multi-label classification task with MultiEURLEX

Error measure: Hamming Accuracy/Zero-one accuracy Uncertainty Quantification: Aggregate over k binary classifiers, entropy of predicted distribution is used for predictive uncertainty (https://proceedings.mlr.press/v70/li17a.html, https://arxiv.org/pdf/2006.11584). Other metrics such as mutual information and variance can be used for the model uncertainty (https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9761166).

J-Dymond commented 1 week ago

Added recent pull request to this issue, so this is implemented now. Could be re-opened when investigating additional models/uncertainty metrics.