langcog / metalab

MetaLab -- Community-augmented meta-analysis
6 stars 8 forks source link

Effect direction in habituation studies: reverse entry or flip during processing? #39

Open christinabergmann opened 4 years ago

christinabergmann commented 4 years ago

I realized recently that we code Habituation studies positive when there is a stronger dishabituation response in the vowels database. This is different from most other types of paradigms (right?) so we either need to make this consistent and flip effect sizes from habituation studies in the code programmatically by checking if something is labelled habituation, or have very clear instructions. @shotsuji and @lottiegasp - Do you have any preference for either? I think each of you solved this in one of the two suggested ways.

lottiegasp commented 4 years ago

In my study I went with consistently coding novelty effect as positive and familiarity effect as negative. This means that in a habituation/familiarity design "novelty" refers to the non-habituated/familiarised condition heard in the test phase, while in a test-only design, "novelty" refers to the less familiar/expected condition in the participant's everyday environment, i.e. in my case the non-native language. That meant that predictions were opposite for the two design types: we predicted an overall novelty preference (positive effect sizes) for habituation/familiarity tasks, and an overall familiarity preference (negative effect sizes) for test-only tasks. In the stats we separated out the two design types so as to not conflate effect sizes going in two directions.

I had mentioned to Sho that we might want to create more explicit instructions on how to code effect size direction. Do you think the consistent approach of positive=novelty/negative=familiarity could work across all meta-analyses? Then we could provide instructions for how to deal with this in the stats if you have predictions in both directions, as I did in my study. Happy to discuss further.

christinabergmann commented 4 years ago

Oh we need to make all this much clearer. We did the same for statistical word segmentation but flipped it, because all other datasets have familiarity=positive and novelty=negative and I do think this should be consistent, e.g. for analyses where we investigate method effects. Alternatively, but this should really be discussed with the board, we programmatically set the majority direction as positive per dataset and if habituation is opposite flip it. But that comes with its own issues...

shotsuji commented 4 years ago

Hi,

Yes, so Lottie's MA and mine are consistent, but different from e.g. Christina's word segmentation one. I think it would make sense to code these consistently across MAs; I wonder whether on the long run we could implement a "flip" possibility in the visualization app? The disadvantage of consistent coding is of course that it will become counter-intuitive for some MAs, for instance if we go with dishabituation/novelty = positive, this might be weird for MAs like word seg where everything will then go down with age.... But I would prioritize consistency. Any preferences for the direction we take?

Am Mi., 18. Nov. 2020 um 10:37 Uhr schrieb Loretta (Lottie) Gasparini < notifications@github.com>:

In my study I went with consistently coding novelty effect as positive and familiarity effect as negative. This means that in a habituation/familiarity design "novelty" refers to the non-habituated/familiarised condition heard in the test phase, while in a test-only design, "novelty" refers to the less familiar/expected condition in the participant's everyday environment, i.e. in my case the non-native language. That meant that predictions were opposite for the two design types: we predicted an overall novelty preference (positive effect sizes) for habituation/familiarity tasks, and an overall familiarity preference (negative effect sizes) for test-only tasks. In the stats we separated out the two design types so as to not conflate effect sizes going in two directions.

I had mentioned to Sho that we might want to create more explicit instructions on how to code effect size direction. Do you think the consistent approach of positive=novelty/negative=familiarity could work across all meta-analyses? Then we could provide instructions for how to deal with this in the stats if you have predictions in both directions, as I did in my study. Happy to discuss further.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab/issues/39#issuecomment-729319866, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD4RHG6KZHILF2BQZYTSPP3SQMQPBANCNFSM4TUMOVWQ .

-- Assistant Professor The University of Tokyo International Research Center for Neurointelligence (IRCN) Personal website: https://sites.google.com/site/tsujish/home Lab website: https://babylab.ircn.jp/ Blog: https://cogtales.wordpress.com/

christinabergmann commented 4 years ago

Jinx, @shotsuji !

shotsuji commented 4 years ago

Yes :) Ok so then it looks like recoding vowels & lang discr makes the most sense?

Am Mi., 18. Nov. 2020 um 18:15 Uhr schrieb Christina Bergmann < notifications@github.com>:

Jinx, @shotsuji https://github.com/shotsuji !

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab/issues/39#issuecomment-729545005, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD4RHG2FI4U6J4K4T56YOTLSQOGEVANCNFSM4TUMOVWQ .

-- Assistant Professor The University of Tokyo International Research Center for Neurointelligence (IRCN) Personal website: https://sites.google.com/site/tsujish/home Lab website: https://babylab.ircn.jp/ Blog: https://cogtales.wordpress.com/

christinabergmann commented 4 years ago

I think this would be least work, but let's talk about majority directions as reference in the long run!

lottiegasp commented 4 years ago

I would be fine with familiarity=positive and novelty=negative if that seems like the least work to switch overall. It would be easy enough to provide some code for users to be able to switch direction of effect sizes in their own study for ease of interpretation and visualisation. So we make clear that if they want their data in MetaLab they code in that direction, but if the majority predicted direction is novelty preference, they can flip all directions in their own publication.

christinabergmann commented 3 years ago

I want to flag the Switch dataset here as well. Lottie, can you check?

lottiegasp commented 3 years ago

I started the process of reversing effect size directions (just having some problems, see https://github.com/langcog/metalab/issues/69) , listing those below that need to be reversed and will check others and add as needed