Draft Module 11: 'Analysis & Modeling'

njlyon0 commented 8 months ago

Summary

Need to create a strong draft of this module so that we can revise and eventually offer this section of the course.

Sub-Tasks

[x] Draft module skeleton
[x] Draft learning objectives
- Consider using a diversity of Bloom's Revised Taxonomy verbs
[x] Draft content
- Could include, text, code, slides, or some combination of those!
[x] Identify companion readings / activities
- For readings, may make sense to divide between readings for before and readings for after
[x] Identify bonus resources
- "Bonus" as in related but external resources that are not a required part of the module but might be useful for students interested in delving deeper
[x] Gather feedback from SSECR design team
- [x] Implement feedback

Resources

SSECR website module page
"Draft Course Objectives" Google Doc
- In Shared Drive for course

njlyon0 commented 6 months ago

Progress!

Fleshed out a pretty solid mixed effects model case study to showcase some of the relevant concepts. Included a nice simulated dataset with the prerequisite nested structure which also set up the architecture (tracked by Git and ignored in advanced) for other modules that benefit for example datasets

njlyon0 commented 6 months ago

Mixed-Effects Fully Drafted

I want to revisit the case study but at least a draft topic covering random intercepts, slopes, and nested random effects is now live in the module

njlyon0 commented 5 months ago

Frequentist vs. Multi-Model Inference Aside Drafted

As I was starting the MMI bit I realized it would just be simpler to address the philosophical difference between the two at the start of the module rather than trying to define MMI in opposition to mixed-effects models after a third of the module's content is already done

njlyon0 commented 5 months ago

MMI Fully Drafted

I may need to revisit this to refine it / add detail but the current draft of the multi-model inference module is not unreasonable.

njlyon0 commented 5 months ago

Meta-Analysis Work Ongoing

Continuing to read into meta-analysis to flesh out that topic of this module. Somewhat slow going as many resources focus more on the use of meta-analysis functions and less on how users need to structure their data going into that process. That feels like the more important bit for our tutorial of meta-analysis so I'm focusing there (at least for now).

The 'meta' R package is promising but their vignette often does not specify arguments so it is difficult to figure out what are hard requirements of the input data file and what are convenience changes to make the vignette functions simpler

scelmendorf commented 5 months ago

@njlyon0 I added a few resources to the task list above if helpful. I think that the structure of data going in will vary depend on what effect size you intend to calculate. But once effect sizes are calculated the syntax should be similarish from there on out. Maybe give 1-2 examples and then refer people to escalc documentation?

njlyon0 commented 4 months ago

Please do attach any resources you have to this issue and I'll integrate them into the module page!

I'm thinking along the same lines as you module structure-wise, just having a hard time finding any solid documentation of what data format(s) are accepted by the effect size calculation functions.

njlyon0 commented 4 months ago

Meta-Analysis Drafted

Drafted meta-analysis section and added Sarah's resources (see below). That section is a little short but we're pressed for time if we do need to do a deep dive on the preceding statistical methods so maybe that's fine. We also need to decide whether to cover this module at all if we're leaning towards a flipped approach (or schedule two--likely consecutive--class sessions to split the load)

New Resources

Slides and other materials from this short course on metaanalysis (using metafor) https://github.com/wviechtb/workshop_2022_ma_esmarconf/blob/master/workshop_meta-analysis.pdf
Conceptual and mathematical linkages between mixed models and metaanalysis (frequentist perspective): https://www.metafor-project.org/doku.php/tips:rma_vs_lm_lme_lmer?s%5B%5D=lme4
Conceptual and mathematical linkages between mixed models and metaanalysis (bayesian perspective): https://mvuorre.github.io/posts/2016-09-29-bayesian-meta-analysis/ and https://solomonkurz.netlify.app/blog/2020-10-16-bayesian-meta-analysis-in-brms-ii/ and https://solomonkurz.netlify.app/blog/bayesian-meta-analysis/ and

scelmendorf commented 4 months ago

@njlyon0 at the risk of making this more work than one wants for what might become a bonus module, I had the thought that changing the two alternate modes of thinking from frequentist vs multimodel to a 3 forms of thinking that is more 'what are you trying to do' orientation would be helpful.

This paper is meaty but I think incredibly helpful in pinning down: what is my goal and then, working backwards from there, what analytical tools you might use to get there (plus some gotcha's of where you really need to be clear about what you have/haven't done in your exploratory analysis to have your p values be meaningful, i.e if you're going to dredge for relationships - go for it, but be clear that's what you did.

I think you could fold what you have into those 3 buckets, putting mixed models under "inference", AIC under "prediction", and maybe adding a 3rd tab that is more on exploration/data dredging (or could just talk about it in general rather than run through a dredge example)?. And then stack the bottoms of each of those pages with targeted resources (i.e. to learn about regularization - look here, blah blah).

njlyon0 commented 1 month ago

Module Framing Change

During the in person week I think we had decided that this module should be only the 'flipped' component (it became very clear talking to the students that many of them are stats experts in their own right already). With that in mind, I tweaked the overview / made a small area at the top of the module to store any files / content that students want to link when the time comes.

So, @scelmendorf I like your idea for an alternate structure to the module but I think that stuff is effectively just bonus content for asynchronous review so I think for now my plan is to just leave it as-is for now. Though this issue will be a nice reminder if we decide to 'actually' offer the module and/or revisit its content!

Also, can you tell me which paper you referenced so I can link it in the 'additional resources'? It sounds cool but I dropped the ball and didn't click the link when it was working and I now get a 404 error 🤕

scelmendorf commented 1 month ago

@njlyon0 paper for you: https://doi.org/10.1002/ecy.3336

njlyon0 commented 1 month ago

Excellent, thanks! I've added it and the website is re-deploying as we speak.

I'm going to go ahead and close this issue but we can definitely revisit the scaffolding for these topics down the line!

lter / ssecr