Inconsistency in Metabin Function with Subgroups

[VanioLjrAntunes]

Note: for support questions on R package meta, please use the R-sig-meta-analysis mailing list. This repository's issues are reserved for feature requests and bug reports.

Classification of issue

Please indicate whether you want to submit a

• [ ] feature request
• [x ] bug report
• [ ] support request (please do not submit support requests here, but use R-sig-meta-analysis)

Summary

I have encountered an inconsistency with the metabin function while conducting a recent meta-analysis involving subgroups. Specifically, I have observed a discrepancy in the weights of individual studies within subgroup A under different scenarios.

1. Forest plot with overall = FALSE, including all three subgroups.
2. Forest plot with overall = FALSE, including only subgroup A.

Despite the overall results being identical, the weights of the individual studies in subgroup A differ between the two situations.

Additional information

Here is the data used in this example data_example.xlsx

Here is the script used

meta.subgroups <- metabin(event.e, n.e, event.c, n.c,
                       data = ma$subgroup_data,
                       method = "MH",
                       method.tau = "REML",
                       sm = "OR",
                       studlab = Author,
                       subgroup = type)

summary(meta.subgroups)

meta.isolated <- metabin(event.e, n.e, event.c, n.c,
                       data = ma$isolated_data,
                       method = "MH",
                       method.tau = "REML",
                       sm = "OR",
                       studlab = Author,
                       subgroup = type)

summary(meta.isolated)

png(file = paste0("subgroups.png"), #specify the file name and the extension file (in this case .png)
      width = 3800, #specify the width in pixels 
      height = 5000, #specify the height in pixels 
      res = 300) #specify the resolution in pixels per inch (e.g. 72 ppi / 300 ppi)

forest(meta.subgroups,
       layout="Revman5",
       sortvar=w.random,
       lab.e=lb.e, lab.c=lb.c,
       label.left= paste("Favors", f.lb.e), col.label.left = "black", ff.lr = "bold",
       label.right= paste("Favors", f.lb.c), col.label.right = "black",
       leftlabs=c("Author", "Year", "Event", "Total", "Event", "Total", "Weight", "OR", "95% CI"),
       leftcols=c("studlab", "Year", "event.e", "n.e", "event.c", "n.c", "w.random", "effect", "ci"),
       rightcols=FALSE,
       pooled.events=TRUE,
       pooled.totals=TRUE,
       random=TRUE,
       common=FALSE,
       diamond.random=TRUE,
       fs.heading=12,
       fs.study = 12,
       fs.hetstat = 12,
       colgap="6mm",
       colgap.forest = "20mm",
       digits=2,
       digits.pval=3,
       col.square="darkblue", col.square.lines="black",
       col.diamond="black", col.diamond.lines="black",
       print.Q = TRUE, print.pval.Q = TRUE, print.tau.ci = TRUE,
       test.overall.random=F,
       overall.hetstat=F,
       just = "center",
       subgroup = T,
       col.subgroup = "black",
       print.subgroup.name = F,
       test.effect.subgroup.random = TRUE,
       overall = F)

dev.off()

png(file = paste0("isolated.png"), #specify the file name and the extension file (in this case .png)
      width = 3800, #specify the width in pixels 
      height = 5000, #specify the height in pixels 
      res = 300) #specify the resolution in pixels per inch (e.g. 72 ppi / 300 ppi)

forest(meta.isolated,
       layout="Revman5",
       sortvar=w.random,
       lab.e=lb.e, lab.c=lb.c,
       label.left= paste("Favors", f.lb.e), col.label.left = "black", ff.lr = "bold",
       label.right= paste("Favors", f.lb.c), col.label.right = "black",
       leftlabs=c("Author", "Year", "Event", "Total", "Event", "Total", "Weight", "OR", "95% CI"),
       leftcols=c("studlab", "Year", "event.e", "n.e", "event.c", "n.c", "w.random", "effect", "ci"),
       rightcols=FALSE,
       pooled.events=TRUE,
       pooled.totals=TRUE,
       random=TRUE,
       common=FALSE,
       diamond.random=TRUE,
       fs.heading=12,
       fs.study = 12,
       fs.hetstat = 12,
       colgap="6mm",
       colgap.forest = "20mm",
       digits=2,
       digits.pval=3,
       col.square="darkblue", col.square.lines="black",
       col.diamond="black", col.diamond.lines="black",
       print.Q = TRUE, print.pval.Q = TRUE, print.tau.ci = TRUE,
       test.overall.random=F,
       overall.hetstat=F,
       just = "center",
       subgroup = T,
       col.subgroup = "black",
       print.subgroup.name = F,
       test.effect.subgroup.random = TRUE,
       overall = F)

dev.off()

Here we have the summary of the meta.objects

summary(meta.subgroups)
            OR            95%-CI %W(common) %W(random)      type
Study A 0.4416 [0.2471;  0.7890]       26.3       13.8 Outcome A
Study A 0.3980 [0.1806;  0.8772]       16.7       12.5 Outcome B
Study A 0.7119 [0.3673;  1.3796]       16.1       13.3 Outcome C
Study B 4.3043 [1.3717; 13.5072]        2.3       10.3 Outcome A
Study C 1.4938 [0.4298;  5.1923]        3.0        9.7 Outcome A
Study C 1.5882 [0.2356; 10.7044]        1.3        6.4 Outcome B
Study C 1.2381 [0.3434;  4.4637]        3.2        9.5 Outcome C
Study D 0.1655 [0.0545;  0.5024]       15.3       10.5 Outcome A
Study D 0.1054 [0.0061;  1.8268]        4.7        3.7 Outcome B
Study D 0.2694 [0.0880;  0.8247]       11.3       10.4 Outcome C

Number of studies: k = 10
Number of observations: o = 1276
Number of events: e = 364

                         OR           95%-CI     z  p-value
Common effect model  0.5598 [0.4211; 0.7441] -4.00 < 0.0001
Random effects model 0.6420 [0.3427; 1.2028] -1.38   0.1666

Quantifying heterogeneity:
 tau^2 = 0.6573 [0.1339; 3.6271]; tau = 0.8107 [0.3659; 1.9045]
 I^2 = 66.2% [33.9%; 82.7%]; H = 1.72 [1.23; 2.40]

Test of heterogeneity:
     Q d.f. p-value
 26.59    9  0.0016

Results for subgroups (common effect model):
                   k     OR           95%-CI     Q   I^2
type = Outcome A   4 0.6069 [0.4061; 0.9072] 19.69 84.8%
type = Outcome B   3 0.4045 [0.2052; 0.7972]  2.76 27.6%
type = Outcome C   3 0.6025 [0.3632; 0.9996]  3.41 41.4%

Test for subgroup differences (common effect model):
                  Q d.f. p-value
Between groups 1.12    2  0.5724

Results for subgroups (random effects model):
                   k     OR           95%-CI  tau^2    tau
type = Outcome A   4 0.8033 [0.2048; 3.1515] 1.6651 1.2904
type = Outcome B   3 0.4517 [0.2042; 0.9991] 0.0532 0.2306
type = Outcome C   3 0.6174 [0.3030; 1.2577] 0.1516 0.3893

Test for subgroup differences (random effects model):
                  Q d.f. p-value
Between groups 0.62    2  0.7336

Details on meta-analytical method:
- Mantel-Haenszel method
- Inverse variance method
- Restricted maximum-likelihood estimator for tau^2
- Q-Profile method for confidence interval of tau^2 and tau
- Continuity correction of 0.5 in studies with zero cell frequencies

summary(meta.isolated)
            OR            95%-CI %W(common) %W(random)      type
Study A 0.4416 [0.2471;  0.7890]       56.0       27.7 Outcome A
Study B 4.3043 [1.3717; 13.5072]        4.8       24.3 Outcome A
Study C 1.4938 [0.4298;  5.1923]        6.5       23.5 Outcome A
Study D 0.1655 [0.0545;  0.5024]       32.6       24.5 Outcome A

Number of studies: k = 4
Number of observations: o = 476
Number of events: e = 192

                         OR           95%-CI     z p-value
Common effect model  0.6069 [0.4061; 0.9072] -2.43  0.0149
Random effects model 0.8033 [0.2048; 3.1515] -0.31  0.7535

Quantifying heterogeneity:
 tau^2 = 1.6651 [0.3322; 27.7299]; tau = 1.2904 [0.5763; 5.2659]
 I^2 = 84.8% [62.0%; 93.9%]; H = 2.56 [1.62; 4.04]

Test of heterogeneity:
     Q d.f. p-value
 19.69    3  0.0002

Results for subgroups (common effect model):
                   k     OR           95%-CI     Q   I^2
type = Outcome A   4 0.6069 [0.4061; 0.9072] 19.69 84.8%

Test for subgroup differences (common effect model):
                  Q d.f. p-value
Between groups 0.00    0      --

Results for subgroups (random effects model):
                   k     OR           95%-CI  tau^2    tau
type = Outcome A   4 0.8033 [0.2048; 3.1515] 1.6651 1.2904

Test for subgroup differences (random effects model):
                  Q d.f. p-value
Between groups 0.00    0      --

Details on meta-analytical method:
- Mantel-Haenszel method
- Inverse variance method
- Restricted maximum-likelihood estimator for tau^2
- Q-Profile method for confidence interval of tau^2 and tau

I saved the forest plots as png to ilustrate better the problem Screenshot A: Scenario 1

Screenshot B: Scenario 2

Environment

• operating system: Windows 11
• R version: R 4.3.1
• Version of R package meta: meta 6.5.0

[guido-s]

Thank you for pointing out this inconsistency.

The problem with subgroup random effects meta-analysis is that two sets of random effects weights exist if separate estimates are allowed for the between-study variance in subgroups:

1. weights based on the overall tau2, i.e., meta.subgroups$tau2
2. weights based on the subgroup-specific tau2, e.g., meta.isolated$tau2 for subgroup A

In your Scenario 1, weights based on the overall tau2 had been printed. This makes sense if the random effects estimate is shown (which is not the case in your Scenario 1).

Now, the subgroup-specific tau2s are used to calculate the percentage weights if argument overall = FALSE.

[VanioLjrAntunes]

Thank you for your response! However, in both scenarios I used the overall = FALSE. Shouldn't both scenarios use the subgroup-specific tau2?

[guido-s]

The subgroup estimates (diamonds for subgroup results in the forest plot) use the subgroup-specific tau2.

However, list element w.random of a meta-analysis object always contains the weights of the overall analysis. This is due to always storing the overall random effects estimate in a meta-analysis object, i.e., list elements TE.random, seTE.random, ...; regardless whether argument random = FALSE or overall = FALSE was used in the creation of the meta-analysis object.

[VanioLjrAntunes]

It makes sense! So what would be the name of the variable which storage the "w.random" version of subgroup-specific tau2?

[guido-s]

There is no variable with this information.

You could calculate this information with the following R code (w.separate = weight; p.separate = percentage contribution to subgroup estimate).

set.seed(1909)
subgr <- as.numeric(sample(100, 10) > 40) + 1
m <- metagen(1:10, runif(10, 0.8, 1.2), subgroup = LETTERS[subgr])
m

w.separate <- rep(NA, length(m$w.random))
p.separate <- rep("", length(m$w.random))
w.separate.w <- rep(NA, length(m$w.random.w))
names(w.separate.w) <- names(m$w.random.w)

for (i in unique(m$subgroup)) {
  sel.i <- m$subgroup == i
  w.separate[sel.i] <- 1 / (m$seTE[sel.i]^2 + replaceNA(m$tau.w[i]^2, 0))
  w.separate.w[i] <- sum(w.separate[sel.i])
  p.separate[sel.i] <- paste0(round(100 * w.separate[sel.i] / w.separate.w[i], 1), "%")
}

data.frame(p.separate, w.separate,
  p.overall = paste0(round(100 * m$w.random / sum(m$w.random), 1), "%"),
  w.overall = m$w.random)

m$w.random.w
w.separate.w

sum(w.separate)
sum(w.separate.w)

sum(m$w.random)
sum(m$w.random.w)

[VanioLjrAntunes]

Perfect! I will implement right away in my scripts. Thank so much, Dr. Schwarzer!