Improve ESR comparative assessment

[tmfrnz]

Via email from @annemariebrook

Susan recently identified a small error in the auto narrative for the ESR comparative assessment.

If the country does not have the full set of ESR data we had originally intended for the narrative to say: “For [Country] a Quality of Life score is not available due to missing data for at least one component of the rights to food, health, education, housing and work. Missing data tells us that [Country] has not submitted some information to the relevant international databases. With more funding, HRMI could investigate the reason for each data gap, and explore ways to help fill it.”

Instead, the narrative is still producing a comparative assessment. E.g. for Fiji (which only has ESR data for 3 of the 5 ESRs) the narrative says “On Quality of Life, Fiji is performing close to average, compared with the other countries in East Asia and Pacific.”

We have discussed this and decided that rather than adopt the original plan, it might be better to come up with a more sophisticated rule for comparing Fiji with other countries in the region.

Option 1: Change wording to something like (new text in bold), e.g. "On Quality of Life rights, when we look across the rights for which we have data, Fiji is performing close to average, compared with the other countries in East Asia and the Pacific."

Note that the comparative assessment in this case would have to be generated by comparing Fiji's score for each available right, and then averaging those assessments, rather than comparing Fiji's average for 3 rights with the region's average for all 5 rights.

Option 2: We could provide a separate comparative statement for each individual right. E.g. “On Quality of Life rights, Fiji is performing better than average on the rights to (e.g. food and work) and close to average on the right to e.g. housing, compared with the other countries in East Asia and the Pacific.

[annemariebrook]

Thanks Timo for creating this issue on GitHub. Keen to hear which of the two options you prefer. Our preference is for Option 1 because it seems closer to the “average” comparative assessment that we provide for the countries with a full set of data. But that’s not a strong view.

[tmfrnz]

Thanks @annemariebrook for following up. I had a quick look at the code and my preference (at least for the short term) would be the original plan as it would be a very quick fix for the error identified. After all this is what the current implementation was built to do.

The other 2 options, and especially option 1, would require significantly more development effort and testing (and possibly also some additional thinking) and I would thus suggest to consider this again at a later stage (that is unless this is higher priority than some of the other urgent issue).

[annemariebrook]

Thanks @tmfrnz. I wouldn't say it is urgent, but the fact that the comparative assessment features on the country snapshot was what led us all to a reasonably strong preference for pursuing Option 1 or 2. Let's discuss at our call this week, as I'm keen to learn more about why you think Option 1 may require some additional thinking.

[tmfrnz]

See #70 for immediate fix

[tmfrnz]

Thanks @tmfrnz. I wouldn't say it is urgent, but the fact that the comparative assessment features on the country snapshot was what led us all to a reasonably strong preference for pursuing Option 1 or 2. Let's discuss at our call this week, as I'm keen to learn more about why you think Option 1 may require some additional thinking.

See here for country snapshots: #60

[annemariebrook]

@tmfrnz I'm a bit unclear on where this issue is at. Do you need further input from us? I am very keen for this issue to be fixed in time for the V3 launch. Thanks!

[tmfrnz]

@annemariebrook at this stage I believe we are all clear to proceed with the suggested option 1 (compare with other countries that we have data for).

The one question I have would be how to handle countries where we only have data for one right? Should we fall back to a text more similar to option 2?

I will let you know should I have any questions or encounter any roadblocks.

[annemariebrook]

@tmfrnz, yes i think that would be consistent. e.g. If two ESR scores are available it might say:

"On Quality of Life rights, when we look across the rights for which we have data, The Marshall Islands is performing close to average, compared with the other countries in the Pacific."

But if only 1 ESR score is available, then it would reference that right by name. e.g. "On Quality of Life rights, Djibouti is performing worse than average on the right to housing compared with the other countries in the Middle East and North Africa".

In the latter case, I would suggest we also add one more sentence just to make it clear that we didn't just forget to mention the other scores: i.e. "Scores for the other Quality of Life rights are not available due to missing data."

Sound OK?

[tmfrnz]

@annemariebrook this sounds good and is ready to implement. Thanks

[tmfrnz]

@annemariebrook a few quick questions to further clarify the future comparative assessment:

• is it correct that we will only consider a country's default assessment standard ?
• should we consider the "income-adjusted" or "global best" benchmark (or maybe even both (as we do in the regular narrative))?

I do not think that the new v3 layout offers any buttons to change those settings for the narrative

[annemariebrook]

@tmfrnz that's a good question. We will discuss. I'm a bit confused by your final comment, ("I do not think that the new v3 layout offers any buttons to change those settings for the narrative"): I wasn't aware of any decision to drop the settings to change benchmarks?

[tmfrnz]

@annemariebrook: my final comment only applies to the narrative, not the charts.

My understanding is that in the new layout we are dropping the main settings panel (floating at the bottom) both on the snapshot (#114) as well as the country details view (ESR tab, #107).

On the snapshot, my understanding is that the user does not get to change any settings, but we could include a benchmark toggle here if you still want this? Not sure though if this should also affect any narrative that we include here?

On the ESR country details view, the settings buttons are now part of each individual chart module (sitting below each chart), suggesting that they only apply to the charts, and not any accompanying narrative that would remain unchanged. While the settings could change for all charts (where relevant) when changed for one chart, I do not think that they should also affect the narrative at the same time (as it cannot be changed individually). Otherwise we would be better off keeping the global settings panel at the bottom...

[annemariebrook]

Hi Timo, Sorry for the delay, I’ve been away so things have slipped. To clarify:

• I don’t have any recollection of discussing dropping the main settings panel from the snapshot page. I think if we did this, users might get frustrated at not being able to change the ESR data view. I suspect the mis-communication around this may have stemmed from the discussion about the print version of the snapshot, which would not include settings information? In any case, I think we need to keep this.

Do you think that keeping the settings panel on the snapshot page creates problems in terms of synchronicity with the ESR country details view, given that the plan was to incorporate the settings buttons as part of each individual chart module?

Best, Anne-Marie

[tmfrnz]

@annemariebrook thank you for clarifying - I have indeed a different recollection of this. Please note that the settings were already dropped in the layouts used for the last user tests.

Anyway, given that we will only have one chart (or group of charts) on the summary page, I would suggest to embed any chart settings right under the chart so we are consistent with the details tab. Will also discuss this with Adina to get her input...

Could you confirm in the meantime if the snapshot view will offer both toggling the benchmark and the assessment standard or just the former? THANKS

[annemariebrook]

@tmfrnz, I think the snapshot should offer both toggling the benchmark and the assessment standard. thank you.

I'm not sure about embedding the chart settings right under the chart. It might be OK but i'm worried about it being so close that people can't snip the chart alone.

[tmfrnz]

@annemariebrook, if keeping the floating settings panel on the snapshot, then we should definitely consider also keeping it on the details tab (and only add any chart-specific settings to the individual charts where required, ie groups, raw data). This would keep the settings consistent across views, simplify the chart settings (removing duplication) and also allow us to apply the settings to the auto-narrative... I will also discuss this with Adina (have a call scheduled with her on Monday). Thanks

[tmfrnz]

@tmfrnz, I think the snapshot should offer both toggling the benchmark and the assessment standard. thank you.

@annemariebrook, as a consequence this means that the comparative assessment will also have to adjust the selected settings

[annemariebrook]

Just to check that I have correctly understood: this issue is not new to Rights Tracker v3.0. It is also relevant on our existing Rights Tracker, right? The issue is that we are not currently clear on the basis of what settings the comparative assessment has been made. Is that correct?

@tmfrnz - I've got this on my list to discuss with Susan.

[tmfrnz]

@annemariebrook in the current rights tracker (v2), the comparative assessment takes the settings correctly into account, but only considers the category scores.

This improvement (where the comp assessment will also look at rights scores) will only be part of the new v3, so this discussion is only relevant for v3.

I am happy for this improved comp assessment also to take the settings into account but just wanted to check if it really needs to...

[annemariebrook]

@tmfrnz Thanks very much for clarifying. I've just discussed this with Thalia, and we would like to add an additional sentence at the end of every ESR comparative assessment that makes it clear which benchmark is being used.

e.g. Using Liberia, which is a country for which the assessment changes with the benchmark, it should say either:.

On Quality of Life, Liberia is performing close to average, compared with the other countries in Sub-Saharan Africa (this comparison is calculated using the ‘income adjusted benchmark’).

OR On Quality of Life, Liberia is performing worse than average, compared with the other countries in Sub-Saharan Africa (this comparison is calculated using the ‘global best benchmark’).

For the countries (like Fiji) that don't have all 5 ESR scores, the same approach would be used. e.g.

On Quality of Life rights, when we look across the rights for which we have data, Fiji is performing close to average, compared with the other countries in East Asia and the Pacific (this comparison is calculated using the '...benchmark'). [Note that in v3 the Fiji comparison should be with the Pacific subregion, not the Asia and Pacific region. I can't remember if we've already discussed that?]

Likewise, for a country with only 1 score, the same approach would be used: e.g.

On Quality of Life rights, Djibouti is performing worse than average on the right to housing compared with the other countries in the Middle East and North Africa (this comparison is calculated using the '...benchmark').

Let me know if this makes sense or if I've not answered one of your questions. Thanks!

[tmfrnz]

@annemariebrook a few follow-up questions:

• Could you confirm: We use the "subregion" (eg Pacific) for comparison only for the Pacific countries. For all other we use the wider "region".
• I assume that we will want to keep the current note about the assessment standard (when not using the default): "Please note: data and commentary are shown for the 'High income' assessment standard. For 'Low and middle income' countries it is best to use the 'Low and middle income' standard instead." (eg https://rightstracker.org/en/country/FJI?as=hi)

[annemariebrook]

@tmfrnz thanks for the excellent questions!

• For ESRs I think we should use the sub-region for comparison in all cases (for low and middle income countries only). For the high-income countries, the ESR comparison should continue to be with the 'other income countries'

• For CPRs, it's a bit more complicated. I'll discuss with Chad and get back to you.

• yes please to keeping the current note about the assessment standard. Thank you!

[tmfrnz]

@annemariebrook thanks for clarifying. I think the focus here only applies to ESR anyway. For CPR-countries we will always have the dimension score so this issues does don't really apply. Also, for CPR we currently compare with all CPR countries and I assume to keep this until further notice (in a separate issue please)

[tmfrnz]

@annemariebrook

On Quality of Life rights, Djibouti is performing worse than average on the right to housing compared with the other countries in the Middle East and North Africa (this comparison is calculated using the '...benchmark').

I wonder if -instead of noting the active benchmark- we should, at least on the details page, just include a sentence for each of the 2 benchmarks (similar to what we have in the main narrative)?

[tmfrnz]

So I suggest we either have (single benchmark)

On Quality of Life, Liberia is performing close to average, compared with the other countries in Sub-Saharan Africa (when scored against the 'Income adjusted' benchmark).

...or something similar to (both benchmarks):

On Quality of Life, Liberia is performing close to average, compared with the other countries in Sub-Saharan Africa, when scored against the 'Income adjusted' benchmark and worse than average when scored against the 'Global best' benchmark.

[annemariebrook]

Agreed! Let's work some more on the auto narrative once we've got v3.0 ready for user testing.

[annemariebrook]

So I suggest we either have (single benchmark)

On Quality of Life, Liberia is performing close to average, compared with the other countries in Sub-Saharan Africa (when scored against the 'Income adjusted' benchmark).

...or something similar to (both benchmarks):

On Quality of Life, Liberia is performing close to average, compared with the other countries in Sub-Saharan Africa, when scored against the 'Income adjusted' benchmark and worse than average when scored against the 'Global best' benchmark.

I agree with your proposal to include a sentence on both benchmarks, at least on the details page. as you propose.