Prosocial MA not appearing in the website

[juliacarbajal]

I noticed there's a recent dataset called Prosocial Agents, for which we have the tabular data and it was also added to the datasets.yaml file, but it is not appearing in the website. I think it's because the link to the table in google drive is missing from the yaml. Was this done on purpose (e.g. the dataset is not ready or the authors asked to wait?) or should I just add the link to the yaml?

[juliacarbajal]

I'm sorry I was wrong, they key to the dataset is there. So it should be appearing on the website but it isn't, so it's definitely a bug. I'll look into it.

[juliacarbajal]

I think I know what's up: the authors of this MA only coded x_1 and x_2 for a within_one participant design. There are no SD, t, F, d or any other measure available to compute the ES. I suspect the processing pipeline discards rows with no effect sizes, is this correct? So if the whole MA has no rows, it doesn't show up in the website. I can contact the authors and ask them to provide the missing data.

[juliacarbajal]

I apologise for this chain of messages. I looked up some of the papers included in this MA and I realise now that there are no SDs because the dependent measure is the number of infants who chose X, and the significance is tested with a binomial test. As far as I understand, the current version of MetaLab only supports traditional meta-analyses for which a d can be calculated, right?

I read here that in this case, the effect size is computed in terms of "relative risk". This would require implementing a new pipeline to process this type of ES, which will probably take a while so I suppose this MA will not be part of MetaLab in the immediate future?

[mcfrank]

IMO it's not at all crazy to convert odds ratios of this type to d...

https://stats.stackexchange.com/questions/68290/converting-odds-ratios-to-cohens-d-for-meta-analysis

Do others have a different thought though? Mike

On Fri, Nov 2, 2018 at 11:17 AM Julia notifications@github.com wrote:

I apologise for this chain of messages. I looked up some of the papers included in this MA and I realise now that there are no SDs because the dependent measure is the number of infants who chose X, and the significance is tested with a binomial test. As far as I understand, the current version of MetaLab only supports traditional meta-analyses for which a d can be calculated, right?

I read here https://stats.stackexchange.com/questions/176847/effect-size-of-a-binomial-test-and-its-relation-to-other-measures-of-effect-size that in this case, the effect size is computed in terms of "relative risk". This would require implementing a new pipeline to process this type of ES, which will probably take a while so I suppose this MA will not be part of MetaLab in the immediate future?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/96#issuecomment-435435640, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGul-vE0wetotstdYLuNWgLXgRbLEPPks5urIxQgaJpZM4YLyCf .

[mllewis]

I think this is very reasonable.

On Fri, Nov 2, 2018 at 2:31 PM Michael Frank notifications@github.com wrote:

IMO it's not at all crazy to convert odds ratios of this type to d...

https://stats.stackexchange.com/questions/68290/converting-odds-ratios-to-cohens-d-for-meta-analysis

Do others have a different thought though? Mike

On Fri, Nov 2, 2018 at 11:17 AM Julia notifications@github.com wrote:

I apologise for this chain of messages. I looked up some of the papers included in this MA and I realise now that there are no SDs because the dependent measure is the number of infants who chose X, and the significance is tested with a binomial test. As far as I understand, the current version of MetaLab only supports traditional meta-analyses for which a d can be calculated, right?

I read here < https://stats.stackexchange.com/questions/176847/effect-size-of-a-binomial-test-and-its-relation-to-other-measures-of-effect-size

that in this case, the effect size is computed in terms of "relative risk". This would require implementing a new pipeline to process this type of ES, which will probably take a while so I suppose this MA will not be part of MetaLab in the immediate future?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/96#issuecomment-435435640, or mute the thread < https://github.com/notifications/unsubscribe-auth/ABGul-vE0wetotstdYLuNWgLXgRbLEPPks5urIxQgaJpZM4YLyCf

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/96#issuecomment-435468566, or mute the thread https://github.com/notifications/unsubscribe-auth/AF32iTh_YZg6oPTqS5onNXjTom0-eIDcks5urJ1_gaJpZM4YLyCf .

[juliacarbajal]

Please correct me if I'm wrong, I'm not so familiar with this kind of analysis, but isn't it necessary to have two treatment conditions to calculate an odds ratio? The kind of experiments and analysis that are reported in this MA go more or less like this: a single group of infants is presented with two choices, A or B. Some number of infants choose A. This is the number reported in the papers, plus a binomial test to evaluate if it's significantly above chance. There are no odds ratios or confidence intervals or anything of the sort. See for example the Results and discussion section of the first experiment in Dunfield & Kuhlmeier (2010), where they have 16 infants, out of which 12 (i.e. 75%) chose option A, and then they performed a binomial test assuming a chance level of 50%. That's all.

Consistently with the data reported in the papers, in this MA the author reported the number of infants who chose A in x_1. Then in x_2 the author coded the number that would be expected at chance (i.e. 50% of the total N). That's all the information we have. Is that enough to calculate an effect size?

[mllewis]

In the past what we've done in these cases is assume there's another group of infants with a mean of chance (.5 in this case) and a standard deviation that's identical to the treatment group.

Molly

On Fri, Nov 2, 2018 at 3:59 PM Julia notifications@github.com wrote:

Please correct me if I'm wrong, I'm not so familiar with this kind of analysis, but isn't it necessary to have two treatment conditions to calculate an odds ratio? The kind of experiments and analysis that are reported in this MA go more or less like this: a single group of infants is presented with two choices, A or B. Some number of infants choose A. This is the number reported in the papers, plus a binomial test to evaluate if it's significantly above chance. There are no odds ratios or confidence intervals or anything of the sort. See for example the Results and discussion section of the first experiment in Dunfield & Kuhlmeier (2010) https://www.jstor.org/stable/pdf/41062241.pdf?casa_token=HEWkweSTIQkAAAAA:Y62jHpMvNAs5i332E9ZnWYIMxreWUctexRb4nt9mjW6lIvRAe417bhMLm6DhgNlrpMQWrN7_EgHZn9C1TXOATEdTpaE6uPreAgePp2bjl8y6TAH42B4, where they have 16 infants, out of which 12 (i.e. 75%) chose option A, and then they performed a binomial test assuming a chance level of 50%. That's all.

Consistently with the data reported in the papers, in this MA the author reported the number of infants who chose A in x_1. Then in x_2 the author coded the number that would be expected at chance (i.e. 50% of the total N). That's all the information we have. Is that enough to calculate an effect size?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/96#issuecomment-435502880, or mute the thread https://github.com/notifications/unsubscribe-auth/AF32iadsC9B9s-B7Q89OJqm8OXLHV-7cks5urLIzgaJpZM4YLyCf .

[shotsuji]

Yes, nice catch Julia. Christina only recently added this MA, and she wrote an email (not github issue) to the metalab group about it, but that was before your time. She and I actually consulted a statistician about it and we came up with exactly the same solution, namely adding chance level as the second group and then later converting to d. So seeing that you all seem to agree on this solution I'd say let's go ahead (also just talked with Christina about it, we're here together at BU, and she agreed).

Am Fr., 2. Nov. 2018 um 17:34 Uhr schrieb Molly Lewis < notifications@github.com>:

In the past what we've done in these cases is assume there's another group of infants with a mean of chance (.5 in this case) and a standard deviation that's identical to the treatment group.

Molly

On Fri, Nov 2, 2018 at 3:59 PM Julia notifications@github.com wrote:

Please correct me if I'm wrong, I'm not so familiar with this kind of analysis, but isn't it necessary to have two treatment conditions to calculate an odds ratio? The kind of experiments and analysis that are reported in this MA go more or less like this: a single group of infants is presented with two choices, A or B. Some number of infants choose A. This is the number reported in the papers, plus a binomial test to evaluate if it's significantly above chance. There are no odds ratios or confidence intervals or anything of the sort. See for example the Results and discussion section of the first experiment in Dunfield & Kuhlmeier (2010) < https://www.jstor.org/stable/pdf/41062241.pdf?casa_token=HEWkweSTIQkAAAAA:Y62jHpMvNAs5i332E9ZnWYIMxreWUctexRb4nt9mjW6lIvRAe417bhMLm6DhgNlrpMQWrN7_EgHZn9C1TXOATEdTpaE6uPreAgePp2bjl8y6TAH42B4 , where they have 16 infants, out of which 12 (i.e. 75%) chose option A, and then they performed a binomial test assuming a chance level of 50%. That's all.

Consistently with the data reported in the papers, in this MA the author reported the number of infants who chose A in x_1. Then in x_2 the author coded the number that would be expected at chance (i.e. 50% of the total N). That's all the information we have. Is that enough to calculate an effect size?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/96#issuecomment-435502880, or mute the thread < https://github.com/notifications/unsubscribe-auth/AF32iadsC9B9s-B7Q89OJqm8OXLHV-7cks5urLIzgaJpZM4YLyCf

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/96#issuecomment-435516008, or mute the thread https://github.com/notifications/unsubscribe-auth/APkTm2CP5ticn5Md7Ol-Bjp9c1w9PYvEks5urLpjgaJpZM4YLyCf .

-- Postdoctoral Researcher Laboratoire de Sciences Cognitives et Psycholinguistique, Paris, France Website: https://sites.google.com/site/tsujish/home Blog: https://cogtales.wordpress.com/

[juliacarbajal]

Oh I see! I apologise, I didn't know this had already been discussed. Thank you all for the clarifications, I understand much better now. In this case I agree this is the best solution.

What needs to be done then?

[christinabergmann]

No worries, Julia, it was before your time and hidden in a very long email full of updates. Thanks for tackling this. I have it on my long term to do list to check whether proportion-based and odds ratio meta-analyses of this and similar datasets arrive at the same conclusion but for now this is a good solution. Let me know when it appears, then I can notify the extremely helpful authors...

[juliacarbajal]

Hi all, as discussed here and also by email with Christina and Sho, I implemented the calculation of d_calc and d_var_calc based on log-odds-ratio for the Prosocial MA. If all goes as expected, this MA should show up next time the website updates. I will close this issue after I see the changes online.

Note that for the moment I wrote this as a special case to apply only to this specific MA, where none of the rows had an SD or t. For other datasets with mixed cases (some rows with SD or t, some without), a decision needs to be made about whether it is reasonable to mix these two approaches or not. I will open a separate issue regarding that.