search

worldbank / institutional-assessment-dashboard

This repository contains the code to create the “global institutional assessment dashboard”
http://w0lxprconn01.worldbank.org:3939/gibd/
5 stars 5 forks source link

UAT | Armenia vs EU only shows 2 families in the overview dynamic benchmarking #222

Closed erwbg closed 10 months ago

erwbg commented 10 months ago

This may be a more general issue with dynamic benchmarking.

Armenia vs EU only shows HRM and Climate. This is too low. Let us please revise what is happening.

kannanv93 commented 10 months ago

@galileukim Looks like there is some data issue with Armenia. All average except the two shown above are all NA.

image
galileukim commented 10 months ago

Thank you for noting this @erwbg and @kannanv93. I have reviewed the data, and Armenia lack of institutional family averages is a feature not a bug.

We have no indicator coverage on the World Justice Project (WJP) and the PEFA families for Armenia in our data. Because WJP indicators are used across families, this means that Armenia will be missing these family averages.

Additionally, please note that in computing the family averages, we are using the restrictive methodology proposed by Rita, that if one of the indicators are missing, the family average is not computed. Should we make this clear in our methodology note?

erwbg commented 10 months ago

@kannanv93 @galileukim thank you so much for this!

it makes sense that the "lack of global institutional family average" is a feature. indeed is perfectly in line with the restrictive methodology for 'global' family averages, that requires all indicators.

however, this does not imply that the "(dynamic) benchmark overview" should be missing. The approach we decided to take here was similar to the static in the sense that some families (SOE, labor) do not get a 'global' family average, but are shown in the dashboard nonetheless.

-> I think this is the important thing to check: that the filter for global dynamic institutional family average is not getting in the way of the benchmarking. @kannanv93 does this makes sense?

Side note: we can double check the "Benchmark_dynamic_indicator" in dev/prod and db_variables. We made some executive and conservative decisions, but the following families should be expected to appear in the overview of the dynamic benchmarking (i.e., "Yes"): Political institutions Social institutions Anticorruption institutions Transparency and Accountability Institutions Justice institutions Public Human Resource Management Institutions Climate Change and Environment Institutions

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

family_order | family_name | Benchmark_static_family_aggregate_download | Benchmark_dynamic_indicator | Benchmark_dynamic_family_aggregate -- | -- | -- | -- | -- 1 | Political institutions | Yes | Yes | Partial 2 | Social institutions | Yes | Yes | Partial 3 | Anticorruption institutions | Yes | Yes | Partial 4 | Transparency and Accountability Institutions | Yes | Yes | Partial 5 | Justice institutions | Yes | Yes | Partial 6 | Public Financial Management Institutions | Yes | No | No 7 | Public Human Resource Management Institutions | Yes | Yes | Partial 8 | Digital and Data Institutions | Yes | No | No 9 | Business environment institutions | Yes | No | No 10 | SOE Corporate Governance | No | No | No 11 | Labor and Social Protection institutions | No | No | No 12 | Service Delivery Institutions | Yes | No | No 13 | Climate Change and Environment Institutions | Yes | Yes | Partial

galileukim commented 10 months ago

@erwbg this is helpful, thank you. please note that all sven institutional family averages mentioned above are included in our dynamic benchmarking. it is just the case that for some countries, some of the institutional family averages cannot be computed due to missingness of indicators. this does not mean we do not have an overview tab in the dynamic benchmarking. for example, even for Armenia, we still have averages computed for the (1) Climate Change and (2) Public HRM institutional families. therefore, there should be an overview tab for it, with only the institutional family averages for (1) and (2).

erwbg commented 10 months ago

thanks, @galileukim! @kannanv93 too

But this is where it's very important to be very clear that there is a global institutional family average (the one that cannot be computed due to missingness of indicators) and a benchmarking institutional family average --the one that is reactive to the base country+comparators.

Take for instance, the Political Family in the Armenia vs EU case --it has plenty of non-missing indicators. As a result, the overview tab should also show the benchmarking institutional family average (in this case for Political) even if WJP indicators are missing. And we should do this following the "even" years filters.

It is very important that we differentiate the global family averages (to download) and the benchmarking family averages (to show in the overview graph, that are reactive to any given base+comparator country sample). And hence we should have more institutional family averages in the overview tab (but not changing anything about the data, the data is fine as is).

@kannanv93 Can we please ensure that the dashboard estimations reflect this?

kannanv93 commented 10 months ago

@erwbg : Please check uat_2 and close this issue

kannanv93 commented 10 months ago

@erwbg : Any problem with this ? Can we close this if tested.

erwbg commented 10 months ago

Thanks, @kannanv93! We have to make changes --it does not seem to be an Armenia-specific issue, but keeping it here for consistency.

Overall, this is working great --with one exception: there are some families for which we should not be a family average in the overview graph (even though individual indicators should still appear in the family-by-family graphs). See pic below.

These 4 families should appear should not appear in the overview graph --but their indicators, should appear in the family-by-family graph (also, in the family-by-family graph, the family-level average of these 4 families should not appear): Digital and Data Institutions Business environment institutions Labor and Social Protection institutions Service Delivery Institutions

image

Happy to clarify or discuss over the phone if easier.

erwbg commented 10 months ago

Works great! Thanks!