Title: Team selection for prediction tasks
Venue: Journal of Combinatorial Optimization
Year: 2016
Main Problem
In this work, authors consider a situation of team formation in which from an available set of experts, each with a certain level of expertise, authors predict the outcome of a continuous variable O using experts’ opinion. For each prediction task, authors gather experts’ opinions and aggregate them by simple linear opinion pooling. The goal is to find a subset of experts with the best performance i.e. a subset of whose aggregated opinion has the least error regarding the actual outcome of O.
Input
A set of Experts, Experts' sequence of past predictions
Output
Best Susbet of experts for a team.
Related Works
Methods for experts’ judgments aggregation include information markets, opinion pooling, Bayesian and behavioral approaches (Chen et al. 2005; Clemen and Winkler 2007).
For information markets, scoring and compensation rules have been introduced to induce truthful forecasts and ensure participation of experts (Othman and Sandholm 2010, etc).
Opinion pooling and Bayesian approaches are mathematical methods for aggregating judgments to obtain accurate probability assessment for an event (Clemen andWinkler 2007, etc).
Lappas et al. (2009), took into account the cost of communication among individuals and presented two approaches for forming a team with minimum, on two different communication cost yet capable of dealing with a defined task, functions.
Proposed Method
First the authors establish that the team formation problem is NP-Hard. Then the authors propose a Tabu search algorithm to solve the Team Formation Problem.
Experimentation
Dataset Used: - The author used artificial datasets which are generated based on calibration and informativeness of experts’ information.
Results: - The results were evaluated into 4 different categories which were based on measures for evaluating quality of expert’s distribution.
Dataset
No Link Available to download Dataset.
Metrics
The author compares different algorithms and heuristics for different cases and plots their average SSE.
Code
Link Unavailable
Gaps of the Work
The authors fail to describe the dataset used in their experimentation process. Moreover, they do not use any other metric apart from SSE to state their results.
The need for quantitative evaluation to be stronger is quite evident in this work because of poor result description.
karan96
commented
2 years ago
Title: Team selection for prediction tasks Venue: Journal of Combinatorial Optimization Year: 2016
Main Problem In this work, authors consider a situation of team formation in which from an available set of experts, each with a certain level of expertise, authors predict the outcome of a continuous variable O using experts’ opinion. For each prediction task, authors gather experts’ opinions and aggregate them by simple linear opinion pooling. The goal is to find a subset of experts with the best performance i.e. a subset of whose aggregated opinion has the least error regarding the actual outcome of O.
Input A set of Experts, Experts' sequence of past predictions
Output Best Susbet of experts for a team.
Related Works
Proposed Method First the authors establish that the team formation problem is NP-Hard. Then the authors propose a Tabu search algorithm to solve the Team Formation Problem.
Experimentation
Dataset No Link Available to download Dataset.
Metrics The author compares different algorithms and heuristics for different cases and plots their average SSE.
Code Link Unavailable
Gaps of the Work