A crowd of 40,000 classifiers leads to an agent-based SWAP model with 80,000 parameters. This is a huge parameter space. The problem factorizes extremely well but still - its a lot of dimensionality. By including an assumption that the volunteers are drawn from a population whose agents' confusion matrix element distribution might be described by a smooth multivariate function (perhaps a Dirichlet distribution?), this parameter volume might be reduced - and on top of this, the noisy inferences about the low experience classifiers could be regularised by our knowledge of the crowd as a whole. There might be gains to be made by reducing the noise in the system in this way. Could be worth thinking about, especially if its a simple additional term in the log posterior being fed to the offline EM engine. (I don't know how the online analysis would implement such a hierarchical model of the crowd.)
A crowd of 40,000 classifiers leads to an agent-based SWAP model with 80,000 parameters. This is a huge parameter space. The problem factorizes extremely well but still - its a lot of dimensionality. By including an assumption that the volunteers are drawn from a population whose agents' confusion matrix element distribution might be described by a smooth multivariate function (perhaps a Dirichlet distribution?), this parameter volume might be reduced - and on top of this, the noisy inferences about the low experience classifiers could be regularised by our knowledge of the crowd as a whole. There might be gains to be made by reducing the noise in the system in this way. Could be worth thinking about, especially if its a simple additional term in the log posterior being fed to the offline EM engine. (I don't know how the online analysis would implement such a hierarchical model of the crowd.)