ebenmichael / augsynth

Augmented Synthetic Control Method
MIT License
143 stars 52 forks source link

Conformal prediction for the example from the vignette returns NAs #104

Closed ashgreat closed 1 month ago

ashgreat commented 1 month ago

I ran the example from the vignette on multisynth and then generated summary using conformal prediction. However, it returns NAs for standard errors, lower bound, and upper bound:

Note: I made minimal coding changes to the example.

library(augsynth)

data <- read.csv("https://dataverse.harvard.edu/api/access/datafile/:persistentId?persistentId=doi:10.7910/DVN/WGWMAV/3UHTLP", sep="\t")

data |> 
  dplyr::filter(!State %in% c("DC", "WI"),
         year >= 1959, year <= 1997) |> 
  dplyr::mutate(YearCBrequired = ifelse(is.na(YearCBrequired), 
                                 Inf, YearCBrequired),
         cbr = 1 * (year >= YearCBrequired)) -> analysis_df

# with default nu
ppool_syn <- multisynth(lnppexpend ~ cbr, State, year, 
                        analysis_df)

summary(ppool_syn, inf_type = "conformal")
#> 
#> Call:
#> multisynth(form = lnppexpend ~ cbr, unit = State, time = year, 
#>     data = analysis_df)
#> 
#> Average ATT Estimate (Std. Error): -0.011  (NA)
#> 
#> Global L2 Imbalance: 0.003
#> Scaled Global L2 Imbalance: 0.019
#> Percent improvement from uniform global weights: 98.1
#> 
#> Individual L2 Imbalance: 0.028
#> Scaled Individual L2 Imbalance: 0.096
#> Percent improvement from uniform individual weights: 90.4    
#> 
#>  Time Since Treatment   Level     Estimate Std.Error lower_bound upper_bound
#>                     0 Average -0.004280701        NA          NA          NA
#>                     1 Average -0.010855744        NA          NA          NA
#>                     2 Average  0.004379480        NA          NA          NA
#>                     3 Average  0.001156364        NA          NA          NA
#>                     4 Average -0.009304510        NA          NA          NA
#>                     5 Average -0.016942492        NA          NA          NA
#>                     6 Average -0.018504802        NA          NA          NA
#>                     7 Average -0.003866051        NA          NA          NA
#>                     8 Average -0.015834891        NA          NA          NA
#>                     9 Average -0.031750413        NA          NA          NA
#>                    10 Average -0.017838072        NA          NA          NA

Created on 2024-08-06 with reprex v2.1.1

Am I missing something?

ashgreat commented 1 month ago

It looks like conformal prediction is available only for the cases with single treated unit. Can someone please confirm if this is correct?

ashgreat commented 1 month ago

It looks like conformal prediction is available only for the cases with single treated unit. Can someone please confirm if this is correct?

Eli confirmed that currently (as of 08/12/2024) they do not have conformal inference procedure implemented with multiple treated units.