Experiment 1 decision and limitations

[a2975667]

Donation limitation

“Discuss the limitations of extracting user behaviors through the donation process from Mechanical Turkers” — 1AC meta (R2)

“The one question I had in the methods and results was about eliciting “true” preferences through donation from mechanical Turk workers. It definitely isn’t intuitive to me that people would donate across multiple different groups ecologically. It sounds like there were reasons to believe that this did work, but the explanations don’t fully make it to clear to me what happened. I understand that many participants did donate, but what I’m wondering about is whether they donated to multiple different charities. It seems like this wouldn’t match well with a survey condition where they may have many more “resources” to spend.” -- R1

“I would encourage the users to think more about the limitations of this domain and potential biases that might impact users’ choice of to where they would donate.” --R2

“I would really like to see a discussion of the limitations of extracting user behaviors through the donation process from mechanical turkers. I would recommend that the authors provide more explanation for this approach and reflect on the limitations of this approach, as well as cite prior works that used this approach to elicit participant behaviors” -- R2

Learning effect limitation

“Discuss whether there may have been a learning effect from participants receiving 2 of the 3 QV versions” -- 1AC meta (2AC)

Justification

“The authors could justify why they have drastically uneven group sizes and why that does not matter in their analysis (Study 1 has 56 subjects in the Likert scale survey group, but 100+ in each of the three QV groups)?” -- 2AC

“Another key methodological question that was not sufficiently justified in my opinion is the decision to give 2 of the 3 QV versions to each QV group participant. Why was this complication added vs. a simpler controlled set up? What about the learning effect and bias to replicate the first version in the second version?” -- 2AC

“I would have appreciated a few more references in the methods section, assuming prior literature was used to inform these decisions (e.g. using a distraction survey; promising to match participants donations; chooing the specific interface features decribed in section 3.3 such as floating the summary panel and disabling out of reach votes; using Student -t distribution vs. alternatives, etc.)” -- 2AC

[a2975667]

Mturk limitation

Should be easy to address. Talk about limited time, lottery, income

Donation limitation

• global (causes) vs local (org)
• add the no effect in the appendix?
• might need to find citations

Learning effect

1. We have the playground
2. We switch the order to minimize the impact of possible learning effects.

Justification

1. we compare each group on the same level, not likert vs qv. Might need to clarify instead of justification

[a2975667]

On learning effect

1. we compare QV and Likert independently. and compare across the differences from the same person.
2. There is no learning effect between QV and donation. these are the two that we compare on.
3. We see most participants used up their budget. They still need to adjust their votes to the budget. It's hard for an individual to transfer directly between two QV. if they have the same strat or even a different strat, they need to redistribute their credits. We found to support that most participants used up their credits. Add to methods: explain why do 2/3 cause we want to see the reasoning. limitation touch on the learning effect.

[a2975667]

On donations: Though there had been several works looking at the context of donation in the CSCW community [https://dl.acm.org/doi/pdf/10.1145/2531602.2531611], we use the donation \textit{behavior} as an approximation of their true beliefs.