search

langcog / metalab

MetaLab -- Community-augmented meta-analysis
6 stars 8 forks source link

metafor update: random effect structure does not work any more #81

Open christinabergmann opened 5 years ago

christinabergmann commented 5 years ago

Right now we have the following random effect structure in our scripts:

random = ~  same_infant_calc | short_cite/unique_row

in server.R

This leads to the following error:

Error in rma.mv(d_calc, d_var_calc, random = ~same_infant_calc | short_cite/unique_row,  : 
Cannot use '~ inner | outer1/outer2' type terms in the 'random' argument.
christinabergmann commented 5 years ago

Hi all, some new info:

  1. Generally, we would want ES- and study-level random effects. So that can be a general approach.
  2. Some MAs have non-independent ES for two reasons: a) The same participants have been measured in multiple conditions (I count first versus second half of the test phase here, too, although we might want to reconsider). b) The same participants have been measured in multiple ways (eg overt and looking responses).

Both cases can be found in the IDS-pref MA.

So far, we treated both reasons the same, but that's not ideal. We should add a way to separate these cases, which I think can be done by checking whether there are differences in response_mode

So we can do a number of things now:

Any preferences?

shotsuji commented 5 years ago

Thanks so much for looking into this, Christina! I think mid/long term switching to option 3 seems ideal, but for the short term what about trying our option 1 and see how it affects fixed effects in the datasets? Should be quick to do.

On Sat, Jun 1, 2019 at 15:41 Christina Bergmann notifications@github.com wrote:

Hi all, some new info:

  1. Generally, we would want ES- and study-level random effects. So that can be a general approach.
  2. Some MAs have non-independent ES for two reasons: a) The same participants have been measured in multiple conditions (I count first versus second half of the test phase here, too, although we might want to reconsider). b) The same participants have been measured in multiple ways (eg overt and looking responses).

Both cases can be found in the IDS-pref MA.

So far, we treated both reasons the same, but that's not ideal. We should add a way to separate these cases, which I think can be done by checking whether there are differences in response_mode

So we can do a number of things now:

  • Overspecify the random effects structure, which can have unwanted consequences (but in most cases will not dramatically affect fixed effects estimates)
  • Have a check that eg more than 5% of ES are affected in a MA and then switch to a different random effects structure (with less, the exercise might be futile anyhow)
  • Look into cluster-robust variance estimators, as implemented in an rma.mv() compatible way in ClubSandwich https://cran.r-project.org/web/packages/clubSandwich/index.html

Any preferences?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/135?email_source=notifications&email_token=AD4RHG34G6ZQMM3ONCQRTBTPYIKX5A5CNFSM4HOK2SK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWW2GCI#issuecomment-497918729, or mute the thread https://github.com/notifications/unsubscribe-auth/AD4RHGZ3YUNVYYEOWAFCVIDPYIKX5ANCNFSM4HOK2SKQ .

-- Assistant Professor The University of Tokyo International Research Center for Neurointelligence (IRCN) Website: https://sites.google.com/site/tsujish/home Blog: https://cogtales.wordpress.com/

christinabergmann commented 5 years ago

Yep, sounds like a plan.

christinabergmann commented 5 years ago

Hi all, I am going ahead and implementing the updated random effects structure as follows: random = 1 | short_cite / same_infant_calc / unique_row

Left to do:

alecristia commented 4 years ago

note that random = 1 | short_cite / same_infant_calc / unique_row + exposure_phase is interpreted as random = 1 | short_cite / same_infant_calc / unique_row / exposure_phase

christinabergmann commented 4 years ago

@alecristia But for the general format, we only want random = 1 | short_cite / same_infant_calc / unique_row right?

alecristia commented 4 years ago

I think so... it's been a while ;)

alecristia commented 4 years ago

by the way, I though we were recommending the use of robumeta (I know 2 of my team members have tried/succeeded in using it), but robu doesn't appear anywhere in our code up here, which seems to be all metafor rma.mv: https://github.com/langcog/metalab2/search?q=same_infant_calc&unscoped_q=same_infant_calc

Nothing to do if that was intentional!

shotsuji commented 4 years ago

Yes I like robumeta - we didn't yet integrate it because it is somewhat less flexible than metafor, so the idea was to use it for specific cases with a lot of dependent rows - which requires significant restructuring of code which nobody has had the time to address. Let's leave the issue open.

Am Do., 30. Juli 2020 um 00:33 Uhr schrieb Alex Cristia < notifications@github.com>:

by the way, I though we were recommending the use of robumeta (I know 2 of my team members have tried/succeeded in using it), but robu doesn't appear anywhere in our code up here, which seems to be all metafor rma.mv:

https://github.com/langcog/metalab2/search?q=same_infant_calc&unscoped_q=same_infant_calc

Nothing to do if that was intentional!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/135#issuecomment-665736291, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD4RHG2OSC5CV2URAFIIURLR6A6MBANCNFSM4HOK2SKQ .

-- Assistant Professor The University of Tokyo International Research Center for Neurointelligence (IRCN) Personal website: https://sites.google.com/site/tsujish/home Lab website: https://babylab.ircn.jp/ Blog: https://cogtales.wordpress.com/

lottiegasp commented 3 years ago

Sharing these resources from ESMAR 2021 in case they help when we get back to this Issue https://osf.io/vyfcj/ https://youtu.be/Q9Nce5pxebY?t=50