Watts-Lab / commonsense-platform

The common sense platform, rate your common sense.
https://commonsense.seas.upenn.edu
1 stars 0 forks source link

View for results and sharing #21

Closed markwhiting closed 1 year ago

markwhiting commented 1 year ago

Need to think more about what this is, but for now we should just report individual commonsensicality (from the paper):

We define $M_j=\sqrt{C_jA_j}$ as commonsensicality for an individual person $j$, where consensus $C_j$ is the fraction of claims for which $j$'s own opinion coincides with the majority opinion, and awareness $A_j$ is $j$’s accuracy, averaged over all claims rated by $j$ when predicting the majority opinion.

markwhiting commented 1 year ago

This might look something like:

You've completed the common sense trial. At any point you can answer more questions by logging in.

Here's your common sense score: ###

This is calculated by comparing your answers to others answers, so it will become more accurate if you answer more questions and it will become more accurate as others answer more questions. If you log in you can continue to see this score as it updates over time.

markwhiting commented 1 year ago

Things to add in he future:

  1. social comparison — how I compare against my friends
  2. society comparison — how I compare with people similar to me

    Answer demographic questions to see how much common sense people like you have

  3. Stretch goal: something like a position on a plot against other people
  4. Stretch goal: something about how rare their beliefs are: e.g., only 1/100 people think "2+2=5 is accurate".
markwhiting commented 1 year ago

From our analysis script, this is how we calculate individual_commonsensicality, which is what we would present to the user.

individual_commonsense <-
  individual_statement_responses %>%
  group_by(statement_number) %>%
  mutate(
    median_I_agree = median(I_agree),
    median_others_agree = median(others_agree)
  ) %>%
  group_by(ResponseId) %>%
  mutate(
    individual_awareness = mean(if_else(median_I_agree == .5, TRUE, others_agree == median_I_agree)),
    individual_consensus = mean(I_agree == median_I_agree),
    individual_commonsensicality = sqrt(mean(individual_awareness) * mean(individual_consensus))
  ) |>
  left_join(statements_data) %>%
  left_join(individual_property_responses) |> 
  ungroup()
markwhiting commented 1 year ago

Can we switch this to a percentage, e.g., 100 * the current result and show % after it. And switch it to something like:

Your common sense score is 63%.

This score reflects the similarity of your beliefs to others, and the accuracy of your perceptions about what others believe.

It might be good to say what percentage of people this score is higher than, e.g.,

You have more common sense than 75% of people.