2016-08-30-Analytic-Strategy-Brainstorm

[andkov]

@ampiccinin @GracielaMuniz @wibeasley @smhofer @Maleeha

Context for the meeting:

• slides for todays meeting: Analytic Strategy development
• original CPA poster (description of Primary Phase of Gen 2 scripts)
• The slides from the meeting on 2016-08-05 with you during which we discussed possible next steps.
• Table 2 - Correlations - correlations between intercepts, slopes, and residuals of bivariate growth models.
• Table 3 - Growth Curves - intercepts, slopes, and covariate loadings of growth models
• Sing model look-up: Update repository, load project in RStudio, and open ./scripts/basic-model-lookup.R

[ampiccinin]

re Static tables: 1) need some additional blank spaces to ensure a space between columns. 2) more inclusive headers (e.g., B (SE) p) will reduce need for ", p=" in each row.

[andkov]

Minutes

Scott

• specific type of model (bivariate); may not survive in the future applications
• complexity is real; need to show and describe it.
• similar studies report SELECTIVELY. They report only things they can make sense of
• what we bring is COMPLEX REALITY
• How do we think about modeling?
• How do we organize people/results?
• Tables may not be published, but offered inofficially
• Not simple results are OK. Need to tell the story anyway

Graciela

• demonstrate inconsistency
• show how different coclusions would have been made, if our vision were restricted
• conclusions are dependent on the starting point
• positive side: how we discovered this inconsistency

Andrey:

• archive online for people to verify the inconsistencies
• trying to tell their own stories

Graciela:

• can't focus only on a single thing / small thing

Scott:

• report restuls as they are
• how does it fit into literature?
• FInkel (very selective on cog variable = criticism)
• Full context : results are mixed. need to be careful how we report
• focus on the tables, not graphs
• how can we move this type of analysis into domains.

Graciela:

• what about the meeting with the orginal driver?
• need to show them all this
• we need to provide the wrap up of the training from Portland
• what is the take home message they can take from all this?

Andrea/Scott

• absence of replication is depressing
• not a happy message
• constant issue in this type of research
• more challanging that

Graciela:

• positive spin: actually, this is an opportunity to demonstrate the limitation of this method:
• opportunity to start thinking in a different way
• let package it positively.

Scott:

• danger of an honest report
• paper :

• results are what they are

  Graciela

• we need a bigger lens to see the bigger picture.

  Will

• reporting false positives
• we're protecting you from looking foolish by giving you a realistic sense about how well your results will replicate

• previous: cherry picking can be motivated by various reasons. SOme measures are just stronger

  Andrea

• can't really change the measures in longitudinal studies

Graciela

• but that's the reason for the positive spin

• still i'd like to have them in a room, and give them our report, for learning purposes.

  Will

• sensitivity analysis?
• should it go into a separaret manuscript? or should we roll it into this one?

• how many results were spurious?

  Graciela

• let's show with one or two examples of how conclusions can be different.

  Scott

• important to wrap up
• in : summary of standard coefficients

• appendix : everything else

  Will

• in single models: med est, med se, se of se.

  NEXT STEP

• single tables
• TITLE: unsavoury truth of replication research.

[andkov]

by @smhofer

from

Hofer, S. M., Gray, K. M., Piccinin, A. M., Mackinnon, A., Bontempo, D. E., Einfeld, S. L., ... & Tonge, B. J. (2009). Correlated and coupled within-person change in emotional and behavioral disturbance in individuals with intellectual disability. American journal on intellectual and developmental disabilities, 114(5), 307-321.

Here is the paper with an example of int-int and slope-slope plots (Fig 1). This would be used for an exemplar set of variables to use for an understanding of correlation magnitude and individual differences in these effects.

Table 3 presents the three sets of correlations for random effects. Here we would present results for only one model—the final one I believe.

[andkov]

Possible title

The Inconvenient Truth: How Complexity of Big Analyses Endangers Replication Research