Discussion: weighting options

[jm-rivera]

Starting a discussion on this @amy-dodd @lpicci96 @Mattie-P

For simplicity, equal weights can be favoured.

Equal weights (or similarly ‘no’ weights) All indicators get assigned the same weight. In other words, the final index could simply be the non-weighted arithmetic average of the normalised indicators.

Pros:

• Simplicity of construction above all.
• No need for a theoretical structure to justify a weighting scheme
• ‘Objectivity’ (very questionable since we are choosing to include some indicators and not others)

Cons:

• Problematic if some indicators are correlated as we would have a form of double counting of the underlying factors that make some indicators be correlated. [can be addressed by averaging the collinear indicators prior to aggregating into the composite index, but this adds complexity]
• In a two-stage aggregation (by dimension and then to form the overall index score), the relative weight of indicators becomes unequal unless all dimensions have the same number of indicators.
• Compensability is difficult to manage. In other words, better performance on a given indicator can compensate for weaknesses on another (proportional to the weight).
• This type of approach is often criticised and it’s hard to defend. Alternative options based on a solid theoretical and methodological framework are generally preferred.

Some alternatives.

All come with complexity cost (not necessarily for building but for explaining)

Unequal weights Indicators are assigned different values for different reasons. These could include coverage (so indicators with full data are assigned a heavier weight than indicators with lots of imputed values), trustworthiness of the source, or expert judgement.

Pros:

• More nuanced approach without necessarily over complexifying the methodology.

Cons:

• Every choice is open for debate/criticism.

Budget Allocation Process Like the above, but we go to a panel of experts to get their take regarding appropriate weights. The basic idea is that each expert gets a series of points to distribute to the indicators and then an average of the experts’ choices is used to create the weights. Another similar (but slightly more complex) is the Analytic Hierarchy Process.

Pros:

• More nuanced approach than equal weights with a more solid justification than unequal weights decided only by the authors.

Cons

• Can be time-consuming and might not yield the expected results.

Data-driven options

Correlation Analysis Correlation analysis can be used to obtain weights in two ways: 1) simple correlation matrix, with the indicator weights being proportional to the sum of the absolute values of that row or column respectively. 2)identifying an indicator to serve as the endogenous criterion to which we compare all other indicators in order to derive their weights.

Pros

• Provides a way to deal with correlated values.
• Provides weights that minimise the risk of ‘double counting’
• It is data-driven

Cons

• Not very useful if correlation is insignificant.
• Option two requires the identification of an endogenous criterion. If such an indicator did indeed exist then not much of a point in using other indicators.
• Correlation is not causality so this approach only shows the behaviour of the indicators with respect to each other (a particular challenge with number 2).

Multiple Linear Regression Analysis could be consider to take this logic one step further and explore causal links, but the assumption that there is strict linearity on these types of issues is very debatable.

Principal Component Analysis (PCA) The idea here is basically a reduction of variables (/indicators). You try to capture the highest possible variance with as few indicators as possible. This is done through a series of linear transformations of the standardised indicators. It can be used in two ways: 1) to set weights or 2) to create new indicators (principal components) that describe your data. Number 1 is easier to communicate and more useful for our purpose so I’ll focus on that one here.

Pros

• Used in many indices, accepted as a solid weight elicitation technique.
• No double counting possible and it is data-driven. Would allow us to use a higher number of indicators or to think less about conflicts created by high correlations.

Cons

• Rather complex. It is difficult to communicate to a non-expert audience.
• Adding/removing outliers can make things unstable. This problem is present in other data-driven methods too though but here they can, in certain cases, have a bigger effect.

Both correlation Analysis and PCA produce weights that are derived from the data. So if the data changes (like if we keep updating this regularly/as new data comes) it can make direct comparisons difficult.

Benefit of the Doubt (a version of data envelopment analysis) The idea here is that most weighting systems will favour certain countries and penalise others. Here, the weights are constructed for each country in such a way as to maximise its performance.

Pros

• No country can argue that the weights are unfavourable to their specific case.
• Very popular as it focuses on identifying the way in which the weights would produce the best result for each country.

Cons

• The literature accuses this approach of not being very neutral (i.e. not all countries are treated equally). Differential weightings can be hard to sell – though there are techniques for arriving at shared weights (but this eliminates some of the pros).
• Could quickly become complex, needing to introduce upper and lower bounds for the weights in order to avoid problematic results.

[amy-dodd]

Thanks for this Jorge - be good to discuss but I'm inclined to keep it as simple as possible for all the reasons we've discussed. At first glance my preferences are probably:

• Equal weights - taking on board the challenges, it is super simple and lends itself to story telling and a discursive approach which suits these purposes well.
• Unequal weights again for simplicity but slightly more sophisticated and addressing some data issues. Weighting by coverage could be useful given we are likely to have some missing data. It would be interesting to try a few versions of this and the first one to see how different it comes out?
• PCA seems interesting but I worry about keeping it live and the explanation needed to go with it. But interesting and could be one we'd want to use more in the future?

Budget allocation process (needs way too much time), correlation and the benefit of the doubt approach don't seem right for our purposes or timeline here.