Fixed coefficients for user defined calendar regressor seem to be ignored

[ThomasLaurentInsee]

I'm trying to use Rjedemtra to compute trading days effect with user specified calendar regressor. In order to limit revisions, I want to estimate the calendar effect once a year and reuse the coefficient with subsequent data. It seems to be what the usrdef.varCoef is meant to do.

So i estimate my coefficient up to 2018

But when I reuse the coefficients (cvscjo2) up to 2019, they seemed to be ignored and the calendar effect are reestimated.

Whenb setting the userdefined variable as Undefined (ex. 3 and 4), it seem to work as intended.

I'm working with the 0.1.6 version.

serie <- ipi_c_eu[,"FR"]

week <- ts(runif(500,-1,1),start = 1990, frequency = 12)
sat  <- ts(runif(500,-1,1),start = 1990, frequency = 12)

calendar <- cbind(week,sat)

# First estimate up to dec 2018

x13_param1 <- x13_spec(transform.function = "None", 
                      x11.mode = "Additive", 
                      transform.adjust = "None", 
                      tradingdays.option = "UserDefined",
                      tradingdays.test = "None",
                      outlier.enabled=TRUE,
                      easter.enabled = FALSE,
                      usrdef.varEnabled = TRUE,
                      usrdef.var = calendar,
                      usrdef.varType = "Calendar",

)

cvscjo<-x13(window(serie, end=c(2018,12)), x13_param1)

summary(cvscjo$regarima)

# coefficients are saved

coeff_cjo <- coefficients(cvscjo$regarima)[c("week","sat")]

# second estimate up to Dec 2019, with fixed coef saved

x13_param2 <- x13_spec(transform.function = "None", 
                      x11.mode = "Additive", 
                      transform.adjust = "None", 
                      tradingdays.option = "UserDefined",
                      tradingdays.test = "None",
                      outlier.enabled=TRUE,
                      easter.enabled = FALSE,
                      usrdef.varEnabled = TRUE,
                      usrdef.var = calendar,
                      usrdef.varCoef = coeff_cjo,
                      usrdef.varType = "Calendar",

)

cvscjo2<-x13(serie, x13_param2)

# Coefficients seems to be ignored
summary(cvscjo2$regarima)

#### Same example but the user variable are set as Undefined

# First estimate up to dec 2018

x13_param3 <- x13_spec(transform.function = "None", 
                      x11.mode = "Additive", 
                      transform.adjust = "None", 
                      tradingdays.option = "None",
                      tradingdays.test = "None",
                      outlier.enabled=TRUE,
                      easter.enabled = FALSE,
                      usrdef.varEnabled = TRUE,
                      usrdef.var = calendar,
                      usrdef.varType = "Undefined",

)

cvscjo3<-x13(window(serie, end=c(2018,12)), x13_param3)

summary(cvscjo3$regarima)

# coefficients are saved

coeff_cjo3 <- coefficients(cvscjo3$regarima)[c("r.week","r.sat")]

# second estimate up to Dec 2018, with fixed coef saved

x13_param4 <- x13_spec(transform.function = "None", 
                      x11.mode = "Additive", 
                      transform.adjust = "None", 
                      tradingdays.option = "None",
                      tradingdays.test = "None",
                      outlier.enabled=TRUE,
                      easter.enabled = FALSE,
                      usrdef.varEnabled = TRUE,
                      usrdef.var = calendar,
                      usrdef.varCoef = coeff_cjo3,
                      usrdef.varType = "Undefined",

)

cvscjo4<-x13(serie, x13_param4)

summary(cvscjo4$regarima)

[AQLT]

Hi, Thanks for reporting this issue. Indeed there was a problem with user-defined calendar regressors, since they are not treated as user-defined variables. I think the PR #88 solves it: in preVar_r2jd, i've used the setFixedCoefficients method, using jcoreg = jsobjct$getCore()$getRegression() and then jcoreg$setFixedCoefficients(paste0("td|r@",varNames), .jarray(coef)).

Below an example. As you can see, using the same span you might have a small (insignificant) difference when fixing a coefficient. I think it comes from a rounding of the last digits.

library(RJDemetra)
set.seed(2021)
serie <- ipi_c_eu[,"FR"]

week <- ts(runif(500,-1,1),start = 1990, frequency = 12)
sat  <- ts(runif(500,-1,1),start = 1990, frequency = 12)

calendar <- cbind(week,sat)
x13_param1 <- x13_spec(transform.function = "None", 
                       x11.mode = "Additive", 
                       transform.adjust = "None", 
                       tradingdays.option = "UserDefined",
                       tradingdays.test = "None",
                       outlier.enabled=TRUE,
                       easter.enabled = FALSE,
                       usrdef.varEnabled    = TRUE,
                       usrdef.var = calendar,
                       usrdef.varType = "Calendar",

)
#> Warning: With tradingdays.option = "UserDefined", the parameters tradingdays.autoadjust, tradingdays.leapyear and tradingdays.stocktd are ignored.

cvscjo<-x13(window(serie, end=c(2018,12)), x13_param1)
summary(cvscjo$regarima)
#> y = regression model + arima (3, 1, 1, 0, 1, 1)
#> 
#> Model: RegARIMA - X13
#> Estimation span: from 1-1990 to 12-2018
#> Log-transformation: no
#> Regression model: no mean, trading days effect(2), no leap year effect, no Easter effect, outliers(3)
#> 
#> Coefficients:
#> ARIMA: 
#>           Estimate Std. Error  T-stat Pr(>|t|)    
#> Phi(1)     0.39437    0.22518   1.751  0.08080 .  
#> Phi(2)     0.13082    0.22872   0.572  0.56771    
#> Phi(3)    -0.31097    0.16625  -1.870  0.06229 .  
#> Theta(1)  -0.65749    0.20849  -3.154  0.00176 ** 
#> BTheta(1) -0.72298    0.04193 -17.244  < 2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Regression model: 
#>               Estimate Std. Error T-stat Pr(>|t|)    
#> week           0.04883    0.21553  0.227   0.8209    
#> sat            0.41042    0.21809  1.882   0.0607 .  
#> LS (11-2008) -12.58946    1.62300 -7.757 1.06e-13 ***
#> AO (5-2011)    9.52331    2.29161  4.156 4.12e-05 ***
#> TC (2-2009)   -7.88766    1.86265 -4.235 2.96e-05 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> 
#> Residual standard error: 2.881 on 324 degrees of freedom
#> Log likelihood = -835.1, aic =  1692, aicc =  1693, bic(corrected for length) =  2.29
coeff_cjo <- coefficients(cvscjo$regarima)[c("week","sat")]

x13_param2 <- x13_spec(transform.function = "None", 
                       x11.mode = "Additive", 
                       transform.adjust = "None", 
                       tradingdays.option = "UserDefined",
                       tradingdays.test = "None",
                       outlier.enabled=TRUE,
                       easter.enabled = FALSE,
                       usrdef.varEnabled    = TRUE,
                       usrdef.var = calendar,
                       usrdef.varCoef = coeff_cjo,
                       usrdef.varType = "Calendar",

)
#> Warning: With tradingdays.option = "UserDefined", the parameters tradingdays.autoadjust, tradingdays.leapyear and tradingdays.stocktd are ignored.

cvscjo_fixed<-x13(window(serie, end=c(2018,12)), x13_param2)
cvscjo2<-x13(serie, x13_param2)
summary(cvscjo_fixed$regarima)
#> y = regression model + arima (3, 1, 1, 0, 1, 1)
#> 
#> Model: RegARIMA - X13
#> Estimation span: from 1-1990 to 12-2018
#> Log-transformation: no
#> Regression model: no mean, no trading days effect, no leap year effect, no Easter effect, outliers(3)
#> 
#> Coefficients:
#> ARIMA: 
#>           Estimate Std. Error  T-stat Pr(>|t|)    
#> Phi(1)      0.3945     0.2245   1.757  0.07981 .  
#> Phi(2)      0.1310     0.2281   0.574  0.56615    
#> Phi(3)     -0.3109     0.1658  -1.875  0.06162 .  
#> Theta(1)   -0.6574     0.2079  -3.162  0.00171 ** 
#> BTheta(1)  -0.7230     0.0418 -17.298  < 2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Regression model: 
#>              Estimate Std. Error T-stat Pr(>|t|)    
#> LS (11-2008)  -12.589      1.614 -7.802 7.84e-14 ***
#> AO (5-2011)     9.523      2.275  4.186 3.63e-05 ***
#> TC (2-2009)    -7.888      1.855 -4.251 2.76e-05 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Fixed other regression effects: 
#>        Coefficients
#> r.week        0.049
#> r.sat         0.410
#> 
#> 
#> Residual standard error: 2.881 on 326 degrees of freedom
#> Log likelihood = -835.1, aic =  1688, aicc =  1689, bic(corrected for length) = 2.255
summary(cvscjo2$regarima)
#> y = regression model + arima (3, 1, 1, 0, 1, 1)
#> 
#> Model: RegARIMA - X13
#> Estimation span: from 1-1990 to 12-2019
#> Log-transformation: no
#> Regression model: no mean, no trading days effect, no leap year effect, no Easter effect, outliers(3)
#> 
#> Coefficients:
#> ARIMA: 
#>           Estimate Std. Error  T-stat Pr(>|t|)    
#> Phi(1)     1.82847    0.13349  13.698  < 2e-16 ***
#> Phi(2)     1.51380    0.13759  11.002  < 2e-16 ***
#> Phi(3)     0.59492    0.08476   7.019 1.18e-11 ***
#> Theta(1)   0.81440    0.14191   5.739 2.08e-08 ***
#> BTheta(1) -0.72145    0.04059 -17.776  < 2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Regression model: 
#>              Estimate Std. Error T-stat Pr(>|t|)    
#> LS (11-2008)  -11.746      1.721 -6.826 3.92e-11 ***
#> AO (5-2011)     8.909      2.190  4.069 5.86e-05 ***
#> TC (2-2009)    -7.441      1.850 -4.021 7.11e-05 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Fixed other regression effects: 
#>        Coefficients
#> r.week        0.049
#> r.sat         0.410
#> 
#> 
#> Residual standard error: 2.902 on 338 degrees of freedom
#> Log likelihood = -867.1, aic =  1752, aicc =  1753, bic(corrected for length) = 2.265

^{Created on 2021-05-04 by the reprex package (v1.0.0)}

[ThomasLaurentInsee]

Merci beaucoup ! Cela devrait beaucoup simplifier l'intégration de Jdemetra dans les comptes trimestriels français (où CJO et CVS sont mis à jour à des fréquence différentes (annuelle vs mensuelle)).

I have the impression that the coefficient are the same, but are rounded differently.

With this solution, are the calendar variables with fixed coefficients treated as calendar variables ? Meaning that they enter in the estimation of model$effects[,"td"]. This serie should be the same (on the common span) in the 3 examples above.

[AQLT]

Sorry for the delay. I'm happy to contribute to the integration of JDemetra+ in the French quarterly national accounts! I confirm that with the new version the user-defined calendar variables are still treated as calendar variables and that the estimates cvscjo$regarima$model$effects[,"tde"] are equals for the 3 examples (up to some insignificant differences due to rounding).