valentinkm
commented
11 months ago Thank you very much for your insightful feedback! Likewise just writing this down for further discussion in our meeting tomorrow. Two approaches have emerged from this in discussion:
-
First idea: Guidelines for evidence-based simulation combined with AC
Firstly, we create a guideline on how to best base your assumptions on evidence
Then: we conduct an AC study where the first person writes a research plan as detailed as possible and both implement the plan of the other person, as well as their own.
We compare the results and draw consequences for our guidelines
(We do a third study together?)
→ This should improve generalization as assumptions are based on evidence and AC process leads to a less subjective Outcome
AC Goal: AC would highlight non-reproducibility even in highly detailed and exact research proposals
-
Second idea: Seperating verbal description and evidence for increased generalizability
Firstly, you create a verbal description of a research plan, conrete enough while not giving hints on preferred parameter selection
The adversary, translates this description into a simulation study, actively selects paremeters and collects evidence for the research plan
Both consult, critique and merge their individual understanding. Possibly, they change the study/ do a combined one
Both interpret results on their own
AC Goal: AC would directly tackle generalizability issues by providing different perspectives on implementing verbal descriptions.
notion-workspace[bot]
Wow this is great work. Just to write this down somewhere though we discuss this tomorrow: I am not exclusively concerned with QRP (though you should definetly think about it it and discuss it in your work) but more with linking verbal descriptions with evidence or more broadly: the issue of generalization. A couple of examples: This method works great on small datasets. Evidence: a simulation where small is 200 people. This methods works fine even with a lot of missingness. Evidence: a simulation with 30% missingness. This method is robust against nonnormality. Evidence: A simulation with Chisq DF = 20.
Or differently understood concepts. A recent example in my own work was that I thought "exploratory" studies are defined by their high probability of generating type I errors, then I found a different paper that was claiming they are "discovery" oriented by virtue of having low prior probability of being true and the recommendation that the type I error rate should be low! The point is two researchers understood two completely different things and hence modeled completely different things but used (almost) the same terminology.