Warning message: Unable to extract a variance-covariance matrix using this `vcov` argument.

[ablackf1]

I am using avg_comparisons to estimate cluster robust standard errors and confidence intervals of the ATT for a multinomial regression object on a dataset derived from the matchit() function. This was working well up until this week when I installed the newest version of marginaleffects. Here is some simulated data to show the warning message I am receiving when passing the matched subclasses to the vcov option:

library(MatchIt)
library(marginaleffects)
library(nnet)
library(sandwich)

set.seed(20240712)

y = sample(c(1,2,3), size = 10000, replace=T, prob = c(0.2, 0.5, 0.3))
x1 = round(rnorm(10000, mean = 75, sd = 3))
x2 = sample(c(0,1,2,3), size = 10000, replace=T, prob = c(0.25, 0.35, 0.20, 0.20))
x0 = sample(0:1, size = 10000, replace = T, prob = c(0.9, 0.1))
dat = data.frame(y, x0, x1, x2, x3)

mm = matchit(x0 ~ x1 + x2, data = dat, exact = 'x1',
             method = 'nearest', distance = 'glm', estimand = 'ATT', replace = F,
             ratio = 1
            )

mdata = match.data(mm)
fit = multinom(y ~ x0, data = mdata)
avg_comparisons(model = fit, 
                variables = "x0",
                vcov = ~subclass,
                newdata = subset(mdata,
                                 x0 == 1),
                wts = "weights",
                comparison = 'lnoravg',
                transform = 'exp',
                hypothesis = 'pairwise')

_Warning message:
   Unable to extract a variance-covariance matrix using this
 `vcov` argument. Standard errors are computed using the
 default variance instead. Perhaps the model or argument is
 not supported by the `sandwich` ('HC0', 'HC3', ~clusterid,
                                  etc.) or `clubSandwich` ('CR0', etc.) packages. If you
 believe that the model is supported by one of these two
 packages, you can open a feature request on Github._ 

I also looked within the comparisons() function to see where the warning was triggered and though I couldn't find it exactly, it looks like within the code there is an assignment of the vcov type but then this does not get passed through to the get_vcov function:

vcov.type <- get_vcov_label(vcov)
vcov <- get_vcov(model, vcov = vcov, type = type, ...)

Thanks for any help or suggestion you can provide.

[vincentarelbundock]

As the error message suggests, I'm not sure this kind of model is supported by the sandwich package. Are you able to produce a square covariance matrix with vcovCL() from that package? If not, then marginaleffects cannot do clustering

[vincentarelbundock]

marginaleffects uses the sandwich package to compute clustered standard errors. Unfortunately, the sandwich package does not appear to support nnet::multinom models.

library(nnet)
library(sandwich)

set.seed(20240712)

y = sample(c(1,2,3), size = 10000, replace=T, prob = c(0.2, 0.5, 0.3))
x1 = round(rnorm(10000, mean = 75, sd = 3))
x2 = sample(c(0,1,2,3), size = 10000, replace=T, prob = c(0.25, 0.35, 0.20, 0.20))
x0 = sample(0:1, size = 10000, replace = T, prob = c(0.9, 0.1))
cl = sample(letters[1:10], 10000, replace = TRUE)
dat = data.frame(y, x0, x1, x2, cl)

fit = multinom(y ~ x0, data = dat)

sandwich::vcovCL(fit, ~cl)
> Error in UseMethod("estfun"): no applicable method for 'estfun' applied to an object of class "c('multinom', 'nnet')"

[ablackf1]

Thank you for answering my question so quickly. The reason why I asked in the first place was because I was and still am producing instances when the warning message does not appear. This is directly from my current R session:

fit0 <- multinom(SumStage ~ HUD_DX, data = mdata_breast_FFSMA, weights = weights)

weights: 12 (6 variable) initial value 85928.069680 iter 10 value 69513.403619 final value 69513.400436 converged

est_f0 = avg_comparisons(fit0, variables = "HUD_DX", vcov = ~subclass, newdata = subset(mdata_breast_FFSMA, HUD_DX == 'HUD at Dx'), wts = "weights", comparison = 'lnoravg', transform = 'exp', hypothesis = 'revpairwise' ) est_f0

The estimates print, with no warning (I am not showing the estimates here). I am not sure why this would happen. They are in fact NOT cluster robust standard errors, as I get the same answer when I comment out the vcov = ~subclass line, but the lack of warning made me think for a while that I did have cluster robust SEs! Now I need to figure out how to get mlogit() to work with avg_comparisons(), as mlogit is supported by sandwich, though this does not seem so straightforward.

[vincentarelbundock]

Oh wow, yes that could be seriously misleading.

Do you have a minimal replicable example you could share? If not publicly here, maybe by email? Ideally it would get:

1. The dataset saved as RDS: `readRDS(mdata_breast_FFSMA, file = "dataset.rds")
2. A script with the fitting call (multinom()) and the marginaleffects command you are trying.

Without a replicable example, it's very difficult to diagnose the issue, unfortunately...

[vincentarelbundock]

FWIW, there's a section in this vignette on mlogit, although I personally find it quite challenging to use with marginaleffects. Very easy to mess things up given the unorthodox data structure.

https://marginaleffects.com/vignettes/categorical.html#mlogit-package

[vincentarelbundock]

Closing because I cannot reproduce. I'll be happy to reopen and look into this if someone can supply a minimal reproducible example. Thanks.

[ablackf1]

I have tried to create a dataset to share where the lack of warning message is reproduced but am encountering the strangest scenario. The lack of warning message seems to be tied to my code file and not my data. For example, if I run the same two lines of code from my Rmd file from earlier this summer, the warning message does not appear, but if I submit them again, under the same open RStudio workspace and data set, in a new Rmd file, the warning message does appear.

This is copy/pasted directly from my console, with the first 2 lines from my new Rmd file and the second 2 lines of code from my original Rmd file.

I cannot understand why this is happening. I have never seen anything like this.

But, the good news is that know of the limitation now and can proceed with my analyses accordingly. I’m sorry I can’t isolate this to a data set, and if you have any suggestions for why different Rmd files would yield different results on the exact same code, I would be all ears.

Thanks for your patience with this! Amanda

fit <- multinom(SumStage ~ HUD_DX, data = mdata, weights = weights)

weights: 12 (6 variable)

initial value 48126.595041 iter 10 value 38874.643976 final value 38873.097394 converged res = avg_comparisons(fit,

• variables = "HUD_DX",
• vcov = ~subclass,
• newdata = subset(mdata,
• HUD_DX == 'HUD at Dx'),
• wts = "weights",
• comparison = 'lnoravg',
• transform = 'exp',
• hypothesis = 'pairwise'
• ) Warning: Unable to extract a variance-covariance matrix using this vcov argument. Standard errors are computed using the default variance instead. Perhaps the model or argument is not supported by the sandwich ('HC0', 'HC3', ~clusterid, etc.) or clubSandwich ('CR0', etc.) packages. If you believe that the model is supported by one of these two packages, you can open a feature request on Github. fit <- multinom(SumStage ~ HUD_DX, data = mdata, weights = weights)

  weights: 12 (6 variable)

  initial value 48126.595041 iter 10 value 38874.643976 final value 38873.097394 converged res = avg_comparisons(fit,

• variables = "HUD_DX",
• vcov = ~subclass,
• newdata = subset(mdata,
• HUD_DX == 'HUD at Dx'),
• wts = "weights",
• comparison = 'lnoravg',
• transform = 'exp',
• hypothesis = 'pairwise'
• )

From: Vincent Arel-Bundock @.> Date: Tuesday, July 16, 2024 at 6:39 PM To: vincentarelbundock/marginaleffects @.> Cc: Amanda Blackford @.>, Author @.> Subject: Re: [vincentarelbundock/marginaleffects] Warning message: Unable to extract a variance-covariance matrix using this vcov argument. (Issue #1166)

  External Email - Use Caution

Oh wow, yes that could be seriously misleading.

Do you have a minimal replicable example you could share? If not publicly here, maybe by email? Ideally it would get:

1. The dataset saved as RDS: `readRDS(mdata_breast_FFSMA, file = "dataset.rds")
2. A script with the fitting call (multinom()) and the marginaleffects command you are trying.

Without a replicable example, it's very difficult to diagnose the issue, unfortunately...

— Reply to this email directly, view it on GitHubhttps://github.com/vincentarelbundock/marginaleffects/issues/1166#issuecomment-2231935047, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4GTVTI5MGPZ76W3AD3PRI3ZMWOHRAVCNFSM6AAAAABKZCS3U6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRHEZTKMBUG4. You are receiving this because you authored the thread.Message ID: @.***>

[vincentarelbundock]

Wow, that's so weird!

Are you using something like renv to use different versions of R or packages in different projects?

[ablackf1]

I know, right?! No, I am not using renv and the R workspace is the same – I am just toggling between two different code files. Something must be specified differently between them, I’m just not sure how or what.

The no-warning may not necessarily be specific to your package, so I appreciate your time! I think we can table this and if I encounter it again I will reach back out.

From: Vincent Arel-Bundock @.> Date: Wednesday, August 14, 2024 at 11:35 PM To: vincentarelbundock/marginaleffects @.> Cc: Amanda Blackford @.>, Author @.> Subject: Re: [vincentarelbundock/marginaleffects] Warning message: Unable to extract a variance-covariance matrix using this vcov argument. (Issue #1166)

  External Email - Use Caution

Wow, that's so weird!

Are you using something like renv to use different versions of R or packages in different projects?

— Reply to this email directly, view it on GitHubhttps://github.com/vincentarelbundock/marginaleffects/issues/1166#issuecomment-2290495154, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4GTVTPUQUCGCXW6W23AD33ZRQOXBAVCNFSM6AAAAABKZCS3U6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGQ4TKMJVGQ. You are receiving this because you authored the thread.Message ID: @.***>