CPR data: ensure handling of incomplete data

[tmfrnz]

Previously CPR data was all or nothing - hence the app's availability check only looks for dimension scores. Now that CPR data can also be partially available, the app should also look at available rights (as is done for ESR data)

[tmfrnz]

From @kchadclay via email

The RightsTracker doesn't quite handle countries with only some missing data on CPRs well. For instance, American Samoa: https://rightstracker-v3-1.web.app/en/country/ASM (Vanuatu, Marshall Islands, and Micronesia are also examples). On the "At a glance" page, it says we haven't produced "safety from the state" data for them, even though we have all of the data but one variable. We probably need to create text like that on the ESRs when we have just some missing data.

Also, the CPR tabs are different when some data are missing. They don't show credible intervals, for instance; just bar graphs (https://rightstracker-v3-1.web.app/en/country/ASM?tab=report-physint). Those bars should have credible intervals

[tmfrnz]

from @annemariebrook via email

I propose the following two paragraphs for the CPR narrative for countries for which we don’t have a full set of scores:

"American Samoa's scores for the rights for which we have data suggest that [a significant number of people are not safe from one or more of the following: arbitrary arrest, torture, disappearance, execution or extrajudicial killing]." (The sentence in brackets should be determined by the average score for those rights for which we do have data. Once implemented I propose we check every country with partial data to check that no unforeseen problems arise from this).

"On Safety from the State, when we look across the rights for which we have data, American Samoa is performing [close to average], compared with other countries in the Pacific ~(this comparison is calculated using the 'Income adjusted' benchmark)~."

(strike-through by me as benchmark irrelevant for CPR data)

[annemariebrook]

Thanks for the correcton Timo!

[tmfrnz]

For handling in narrative see #141

[tmfrnz]

Here an additional thought:

For the CPR comparison, we use the confidence interval as the relevant range for the comparison with the average, mean score of the reference group. However I do not think that the methodology would allow to simply average the confidence intervals across rights (after all the confidence intervals of the dimension scores are also not the average of the individual rights scores).

@kchadclay could you clarify/confirm?

[tmfrnz]

Here an additional thought:

For the CPR comparison, we use the confidence interval as the relevant range for the comparison with the average, mean score of the reference group. However I do not think that the methodology would allow to simply average the confidence intervals across rights (after all the confidence intervals of the dimension scores are also not the average of the individual rights scores).

@kchadclay could you clarify/confirm?

To clarify: we have three outcomes of the comparison

1. Better: lower bound is larger than reference (for ESR: score - 5)
2. Worse: upper bound is smaller than reference (for ESR: score + 5)
3. Close: reference is within lower and upper bound (for ESR: within score +/- 5)

The reference is the average of the mean scores of the reference group

[annemariebrook]

Thanks very much Timo for your careful consideration of this.

Chad, Thalia and I discussed this issue this morning and we decided, given the complications, and the fact that there are very few countries for which we have incomplete CPR scores, that we should NOT include a comparative assessment for countries in this circumstance.

Instead – for countries with some, but incomplete Safety from the State or Empowerment rights scores – we would like to replace the existing text as follows:

Existing text: Safety from the State and Empowerment data have not yet been produced for American Samoa. We would like to expand our data collection to the whole world as soon as possible. To learn more about how you can help, please visit this page on the HRMI website.

Proposed new text: A full set of [Safety from the State] data has not yet been produced for [American Samoa]. We hope to collect more complete data in future years.

How does that sound?

[tmfrnz]

Sounds good - thank you for confirming

[tmfrnz]

@annemariebrook deployed to v3.1 site for you to review

Please make sure to also review and update all "no Data: translations (branch "cpr-comp-assessment-141", field: "hrmi.components.CountryNarrative.compAssessmentCPR.noData")

[annemariebrook]

Hi @tmfrnz, this now looks good for countries that have a partial set of data within one CPR category. However, it should not be applied to all countries with missing CPR data. There are two other cases:

1. We need a slightly different sentence for categories where there is no data at all. e.g. for the Marshall Islands, we have partial data in the Empowerment category, but no data in the Safety from the State category. see https://rightstracker-v3-1.web.app/en/country/MHL?tab=snapshot

For those where there is NO data (rather than partial data) I propose we just modify the sentence, so it would say:

Safety from the State data has not yet been produced for the Marshall Islands. We hope to be able to fill this gap in future years.

instead of: A full set of Safety from the State data has not yet been produced for the Marshall Islands. We hope to collect more complete data in future years.

Just to clarify - this new sentence should only be used for countries where we have run the survey, and still have no data (due to insufficient responses). Right now this is only applicable to the Pacific region (as everywhere else we have run the survey we have collected a full set of data).

2. For countries where we have not run the survey at all the text should be the same as on the live site currently. e.g. for Canada the live site says: "Safety from the State and Empowerment data have not yet been produced for Canada. We would like to expand our data collection to the whole world as soon as possible. To learn more about how you can help, please visit this page on the HRMI website." This should not be changed.

[annemariebrook]

@annemariebrook deployed to v3.1 site for you to review

Please make sure to also review and update all "no Data: translations (branch "cpr-comp-assessment-141", field: "hrmi.components.CountryNarrative.compAssessmentCPR.noData")

@tmfrnz I propose to delay asking Sarah to update the translations until the new sentence has also been added, so she can do both at once.

[tmfrnz]

To recap: we will distinguish 3 cases of missing data

1. Some rights scores missing for one of the two CPR categories (eg ASM missing one Safet from the State score)
2. All rights scores missing for one CPR category but (at least some) scores available for the other CPR category (eg FSM missing all Empowerment scores) << this would require the new sentence
3. No CPR data at all (eg TUV)

[annemariebrook]

No that's not correct. Your second two cases above (FSM for Empowerment, and TUV for Empowerment and Safety from the Sate) are actually the same. In these cases all rights scores are missing from a CPR category for a country where we ran the survey.

Here's my recap of the 3 cases of missing data:

1. Some rights scores missing for one of the two CPR categories (eg ASM missing one Safety from the State score). v3.1 is currently correctly set up for this case.

2. All rights scores missing for a CPR category in a country where we have run the survey (e.g. FSM for Empowerment or TUV for both CPR categories). For this case, the new sentence needs to be added - e.g. for the FSM example it would say.

Empowerment data has not yet been produced for Micronesia (Federated States). We hope to be able to fill this gap in future years.

3. All rights scores missing for a CPR category in a country where we have not run the survey. e.g. Canada. In this case, v3.0 is correct, but v3.1 is not. So we need to revert to the v3.0 code. See Canada example here.

[tmfrnz]

Thanks @annemariebrook, I will fix the 3rd case (no CPR data at all) however the problem for the 2nd case is that it is impossible to know if a survey was run in a country if there is no CPR data (eg TUV). The best we can do is infer from the presence of "(at least some) scores" which I am suggesting to use as a proxy.

That is except we are either

• (a) adding a column to the countries table (explicitly storing for each country if the CPR survey was run) or
• (b) adding special pseudo-scores (eg "-1") for those rights where a survey was run but that did not result in a proper score.

Either way both options might need some additional considerations and also require more effort so I don't think they are feasible for the pending release.

Instead, at least for now the text blocks might need to be derived from the presence (or absence) of CPR scores

[annemariebrook]

Thanks @tmfrnz. All the countries where we ran the survey but collected insufficient data are in the Pacific. So I think you can just use the "Pacific" sub-region as a proxy for where the survey was run. I've checked, and there are no countries in the Pacific where we didn't run the survey.

So for the 2nd case, the sentence would be included if the country is in the Pacific sub-region, and there are no scores in the relevant CPR category.

This may not be sufficient in future years, but it's definitely good enough for this year.

Let me know what you think.

[tmfrnz]

It would not be good practice but sure we could just check for the sub-region

[tmfrnz]

This has now been implemented and deployed to our v3.1 dev site.

1. Some CPR rights but not all: https://rightstracker-v3-1.web.app/en/country/ASM
2. No CPR rights but survey run (Pacific subregion or any CPR rights scores present): https://rightstracker-v3-1.web.app/en/country/TUV?as=core
3. No CPR rights and survey not run: https://rightstracker-v3-1.web.app/en/country/CAN?as=hi

[annemariebrook]

Thanks @tmfrnz. The first two cases look good. The Canada example is still not correct. It should include the same text that is currently on the live site. e.g. https://rightstracker.org/en/country/CAN?as=hi

[tmfrnz]

Apologies! How about now? Note that this is however only ever referring to the relevant CPR dimension and not both (this way we won't have the same text duplicated on the snapshot and more targeted on the CPR details tabs)

[annemariebrook]

Looks good, thanks. Would you mind removing the TODO's from the translations for the 3rd case (e.g. Canada), so that Sarah doesn't get confused? Thanks!

I think this issue is now complete with the exception of translations for the 2nd case. Is that right?

[tmfrnz]

I have left the TODOs for the 3rd case as I would like you and Sarah to review this case also as I have changed it from referring to both dimensions to only one CPR dimension

Especially the FR version where the dimension names were previously both written out in the message and lowercase

"Les données pour l'intégrité physique et l'autonomisation n'ont ..."

that has now been changed to

"Les données pour l'{dimension} n'ont ..."

using the (uppercase) name of the dimension.

Regardless, I will now merge these changes into the v3.1 branch so please make all text edits there.

[annemariebrook]

@sarahbichan, would you mind reviewing the translation files in the v3.1 branch for all remaining TODOs? I'm not sure i've passed them all on to you so this is probably the best way of checking we haven't missed any. Thank you!

[annemariebrook]

@sarahbichan - can you please provide an update on where this is at? Thank you.

[sarahbichan]

Not started yet as I'm focusing on things that need to be done before launch. This won't be looked at till next week. Does this also need to be completed before launch?

[annemariebrook]

@sarahbichan yes, definitely before launch, and ideally before Thursday this week, because that's when we will be making v3.1 will be available to journalists. I think it should be a pretty quick job.

[sarahbichan]

I'll look at this now.

[sarahbichan]

With a Ctrl-F search, it looks like there's only one in EN: "hrmi.settings.scale.intro": "TODO: scale intro". Please advise what you'd like to do with this.

No TODOs for FR or PT, but 4 in ES so I'll check those now.

[annemariebrook]

I'm not sure what hrmi.settings.scale.intro is.

I don't understand why you are not finding TODO in FR or PT because this is definitely showing up on the rights tracker. e.g. go to this page: https://rightstracker-v3-1.web.app/fr/country/WSM?as=core and read the narrative on the Quality of Rights scores.

[sarahbichan]

TODOs in ES done. There seem to be places in the FR and PT (e.g. Adult Survival: "hrmi.indicators-raw.adult-survival") where there is still English, but it's not marked with a TODO. How should I treat these fields?

[sarahbichan]

Mmmm. Very strange. When I searched for the text rather than the TODO, it showed up. I'll have to do a visual scan rather than a search. Bear with me.

[annemariebrook]

TODOs in ES done. There seem to be places in the FR and PT (e.g. Adult Survival: "hrmi.indicators-raw.adult-survival") where there is still English, but it's not marked with a TODO. How should I treat these fields?

I think this is part of the data source updates. We had an email exchange about that last week. This will need to be translated, but i said I donn't consider it essential to do pre-launch if you don't have time.

[sarahbichan]

Definitely don't have time, sorry!

[sarahbichan]

Just FYI @tmfrnz , these two fields appear in ES, FR, PT in the reverse order to EN. Not sure if it matters, but it did make it a little confusing when reviewing the translations. (So data but survey comes first for the translations). hrmi.components.CountryNarrative.compAssessmentCPR.noDataButRights" "hrmi.components.CountryNarrative.compAssessmentCPR.noDataButSurvey"

[sarahbichan]

Hopefully the TODOs are all taken care of now.

[annemariebrook]

Awesome, thanks @sarahbichan. @tmfrnz please let us know when you've synced the latest changes.

[tmfrnz]

@sarahbichan are you sure you are looking at the right branch? ("dev-v3.1"). As noted here there are quite a few TODOs still https://github.com/HumanRightsMeasurementInitiative/hrmi-dataportal/issues/141#issuecomment-644162725

[tmfrnz]

With a Ctrl-F search, it looks like there's only one in EN: "hrmi.settings.scale.intro": "TODO: scale intro". Please advise what you'd like to do with this.

No TODOs for FR or PT, but 4 in ES so I'll check those now.

Please ignore this as it was decided to to include a text here (this is for the categories/rights scale toggle available only on the country overview page