Display-Lab / knowledge-base

Precision Feedback Knowledge Base
https://display-lab.github.io/knowledge-base/
Creative Commons Attribution Share Alike 4.0 International
1 stars 1 forks source link

message templates need to accurately describe performance levels that are equal to a comparator #194

Closed zachll closed 5 months ago

zachll commented 6 months ago

Motivation: The top performer message template indicates that the recipient's performance level is above the level of the 90th percentile benchmark. However, the benchmark and the recipient's performance level are commonly equal, for example at 100%. This is likely a problem for the 75th percentile message too.

For example: "Your performance was 100.0% (28/28 cases), above the top 10% peer benchmark, which was 100.0% this month."

Proposed solutions:

  1. Create a new message that is more vague, like "at or above the top 10% peer benchmark". However, this makes it clear that we don't know what the value is.
  2. Develop message templates and causal pathway variants that say "equal to" and which match new preconditions. This would likely require a lot of work, but would make the system sound a lot more intelligent.
  3. Find a new wording that doesn't directly address the equal to vs above characteristic, somehow
zachll commented 5 months ago

Recommended solution:

Revise all templates that say "above" a comparator to state the performance level of the comparator in a separate and-ed phrase like this:

"You are a top performer this month for the measure [measure name]. Your performance was [recipient performance level] and the top 10% peer benchmark was [comparator performance level]."

zachll commented 5 months ago

Next steps:

  1. Review message templates in CEDAR to consider changing phrasing from "above" or "below" to two independent phrases with an "and" (see example above) so that the sentence is not incorrect when the performance levels of the recipient and comparator are equal
  2. Propose new wording in this issue for each case identified
yidanedencao commented 5 months ago

Message templates in CEDAR have been updated with new wording.

zachll commented 5 months ago

Changes have been saved in the zachll-patch-1 branch