Open lottiegasp opened 3 years ago
@lottiegasp can you provide a screenshot or more description of the issue? I'm not yet very familiar with the effect size calculation, but if I can see an example of what's not working, I may be able to assist. Thanks!
For example, NaturalSpeechSegmentation_Neuro should have 26 effect sizes, but on the MetaLab website only 6 show up. These are the six where means and standard deviations are included (rows 17-22, columns BB to BE), but 20 should be able to be calculated from F values (column BI) (and there are 8 rows where no effect size could be calculated because none of means, SDs, t or F are available, this is normal).
The effect size calculation code is meant to include an if-then structure such that effect sizes are calculated from means and SDs if they are available, but if not they are calculated from the included t or F values (see bolded code below. this is the code I used in my meta-analysis, which was based on the code from a previous MetaLab study). But there must be an error whereby the effect sizes are not being calculated from t or F values
for(line in 1:length(db$n_1)){ if(db[line,]$participant_design == "within_two"){ if (complete.cases(db[line,]$x_1, db[line,]$x_2, db[line,]$SD_1, db[line,]$SD_2)) {
pooled_SD <- sqrt((db[line,]$SD_1 ^ 2 + db[line,]$SD_2 ^ 2) / 2)
db[line,]$d_calc <- (db[line,]$x_2 - db[line,]$x_1) / pooled_SD
db[line,]$es_method <- "group_means_two"
} **else if (complete.cases(db[line,]$t))** {
#Dunlap et al., 1996, p.171
wc <- sqrt(2 * (1 - db[line,]$corr))
db[line,]$d_calc <- (db[line,]$t / sqrt(db[line,]$n_1)) * wc
db[line,]$es_method <- "t_two"
} **else if (complete.cases(db[line,]$F))** {
wc <- sqrt(2 * (1 - db[line,]$corr))
db[line,]$d_calc <- sqrt(abs(db[line,]$F) / db[line,]$n_1) * wc
db[line,]$es_method <- "F_two"
}
#Next step: effect size variance (needed for weighting the effect sizes)
#Lipsey & Wilson (2001)
if (complete.cases(db[line,]$n_1, db[line,]$d_calc)) {
db[line,]$d_var_calc <- (2 * (1 - db[line,]$corr) / db[line,]$n_1) +
(db[line,]$d_calc ^ 2 / (2 * db[line,]$n_1))
}
}else if(db[line,]$participant_design == "between"){ if (complete.cases(db[line,]$x_1, db[line,]$x_2, db[line,]$SD_1, db[line,]$SD_2)) {
pooled_SD <- sqrt(((db[line,]$n_1 - 1) * db[line,]$SD_1 ^ 2 + (db[line,]$n_2 - 1)
* db[line,]$SD_2 ^ 2) / (db[line,]$n_1 + db[line,]$n_2 - 2))
db[line,]$d_calc <- (db[line,]$x_2 - db[line,]$x_1) / pooled_SD
# Lipsey & Wilson (2001)
} else if (complete.cases(db[line,]$t)) {
# Lipsey & Wilson, (2001)
db[line,]$d_calc <- db[line,]$t * sqrt((db[line,]$n_1 + db[line,]$n_2) /
(db[line,]$n_1 * db[line,]$n_2))
} else if (complete.cases(db[line,]$F)) {
# Lipsey & Wilson, (2001)
db[line,]$d_calc <- sqrt(abs(db[line,]$F) * (db[line,]$n_1 + db[line,]$n_2) /
(db[line,]$n_1 * db[line,]$n_2))
}
if (complete.cases(db[line,]$n_1, db[line,]$n_2, db[line,]$d_calc)) {
#now that effect size are calculated, effect size variance is calculated
db[line,]$d_var_calc <- ((db[line,]$n_1 + db[line,]$n_2) / (db[line,]$n_1 *
db[line,]$n_2))
+ (db[line,]$d_calc ^ 2 / (2 * (db[line,]$n_1 + db[line,]$n_2)))
}
}else if(db[line,]$participant_design == "within_one"){ if (complete.cases(db[line,]$x_1, db[line,]$x_2, db[line,]$SD_1)) { db[line,]$d_calc <- (db[line,]$x_2 -db[line,]$x_1) / db[line,]$SD_1 db[line,]$es_method <- "group_means_one" } else if (complete.cases(db[line,]$t)) { db[line,]$d_calc <- db[line,]$t / sqrt(db[line,]$n_1) db[line,]$es_method <- "t_one" } if (complete.cases(db[line,]$n_1, db[line,]$d_calc)) { db[line,]$d_var_calc <- (2/db[line,]$n_1) + (db[line,]$d_calc ^ 2 / (2 * db[line,]$n_1)) } } }
@christinabergmann and @shotsuji we figured out that in the case of Fleur's neuro segmentation dataset, effect sizes aren't being calculated because there is no corr values in the spreadsheet, and the code is such that if corr values exist, new values are imputed based on existing values. But if no corr values originally exist there is no alternate corr imputation method, see the code below
set.seed(111)
dataset_data = dataset_data %>% mutate(corr = abs(corr)) %>% mutate(corr = ifelse(corr > .99 | corr < .01, NA, corr))
if (all(is.na(dataset_data$corr))) { dataset_data$corr_imputed <- NA _## corrimputed=NA if no original corr values are in the dataset } _else { dataset_data$corr_imputed <- datasetdata$corr %>% Hmisc::impute(fun = "random") %>% as.numeric() }
} else if (participant_design == "within_two") { if (is.na(corr) & complete(x_1, x_2, SD_1, SD_2, t)) {
corr = (SD_1^2 + SD_2^2 - (n_1 * (x_1 - x_2)^2 / t^2)) / (2 * SD_1 * SD_2)
}
if (is.na(corr) | corr > .99 | corr < .01){
#if correlation between two measures is not reported, use an imputed correlation value
#we also account for the observation that some re-calculated values are impossible and replace those
**corr <- corr_imputed** **_## if no original corr values are in the dataset as above, then corr=NA_**
}
from line 72: } else if (complete(t)) { wc <- sqrt(2 * (1 - corr)) _## corr=NA, so wc=NA, dcalc=NA, no effect size is calculated** d_calc <- (t / sqrt(n_1)) * wc #Dunlap et al., 1996, p.171 es_method <- "t_two" } else if (complete(f)) { wc <- sqrt(2 * (1 - corr)) _## corr=NA, so wc=NA, dcalc=NA, no effect size is calculated* d_calc <- sqrt(f / n_1) wc es_method <- "f_two" }
Should we perhaps adjust the code so that if no corr values exist in the dataset, then the corr values across all the MetaLab datasets are used to impute corr values in the dataset?
Relatedly, I noticed that the MetaLab script's current method for imputing missing corr values is to randomly select from other existing values in the dataset (see in italics above). For my study I imputed from a normal distribution using the median and variance of existing corr values and then adjusting the range to [-1,1]. Rabagliati et al. (2018) imputed it as the mean weighted by sample size. Are you happy with current method for imputing corr, or should we discuss what is currently considered to be the best practice?
Oh, good catch.
How about the following:
Thanks @christinabergmann
@erikriverson as a quick solution, are you able to work on some code that for each row of data extracts corr values from all MetaLab datasets with the same methods, and similar age (maybe where mean_age is +/- 15 days of the mean_age of the row in question that is missing a corr value?) into an object (named below _CORRS_FROM_ALLDATASETS) and then imputes a corr value in the same way?
set.seed(111)
dataset_data = dataset_data %>% mutate(corr = abs(corr)) %>% mutate(corr = ifelse(corr > .99 | corr < .01, NA, corr))
if (all(is.na(dataset_data$corr))) { dataset_data$corr_imputed <- _CORRS_FROM_ALLDATASETS** %>% Hmisc::impute(fun = "random") %>% ## same imputation method as below, when there are corr values in the dataset as.numeric() } else { dataset_data$corr_imputed <- dataset_data$corr %>% Hmisc::impute(fun = "random") %>% as.numeric() }
Let me know if this makes sense.
Then at a later time I can work on 2. and 3. as suggested by Christina. This will likely be in Feb after the conference.
It looks like rows with effect sizes on the website only show up if they have been calculated from means and SDs, not from F or t values. (I have checked that this is the case in Fleur's and my datasets)
Not sure if this is a problem with the script content or its deployment so assigning @christinabergmann and @erikriverson. Let me know if I can help