wbnicholson / BigVAR

Dimension Reduction Methods for Multivariate Time Series
56 stars 17 forks source link

cv.BigVAR assertion code incorrectly written for single lambda parameter #24

Open extrospective opened 3 years ago

extrospective commented 3 years ago

My code reads:

Model1 <- constructModel(mdata, p=2, "OwnOther",ownlambdas=TRUE, gran=c(1), intercept=FALSE)
Model1Results = cv.BigVAR(Model1)

An error results:

Error in if (object@Granularity[2] == 1) { : missing value where TRUE/FALSE needed

The cause of this error has been determined as follows:

BigVARObjectClass.R around line 592 attempts to check for gran having length 1, but does so incorrectly:

        if(object@Granularity[2]==1){
            stop("only one penalty parameter; run BigVAR.est instead of cv.BigVAR")
        }

When I invoke with a single parameter for gran, this code is intended to tell me I have one penalty parameter, but instead the condition check fails. Based on other code in the repo, this one line should be corrected to:

if (length(object@Granularity)==1) { ... }

Checking the length rather than dereferencing second element of an array which may have length 1. As this fix is made, consider whether ==1 or <=1 is more appropriate.

This is a minor issue, now that it is understood and documented here. The seeming workaround is to call BigVAR.est rather than cv.BigVAR when one lambda is supplied. However, as I write below, there are reasons I was trying to use cv.BigVAR, and these are workarounds already.


I could fix and push, but I have run into some other hiccups on this code path. So I can use this ticket also to explain what I am doing and issues I encounter.

a. I want to show that BigVAR and VAR are equivalent at lambda=0, before starting to vary lambda. This is not directly possible since lambda=0 is explicitly allowed (not sure why) and I can approximate with lambda = 0.00000001. FYI. I think it would be nice to support lambda=0. Is there a reason why lambda=0 cannot be directly supported?

b. But when I use BigVAR.est (to test lambda=0.0000001), I encounter problems calculating FEVD with frequencyConnectedness::genFEVD. The latter is designed to work with BigVAR, but explicitly checks the class returned from the fit function (cv or est). cv.BigVAR returns a class which checks out with frequencyConnectedness. But BigVAR.est returns a (fairly limited) list type, and frequencyConnectedness refuses to report the FEVD since the class type assertion fails.

My sense is that this may be a BigVAR issue because BigVAR.est is not building out a full structure similar to cv.BigVAR, and therefore I cannot then use frequencyConnectedness. But I defer to the author in determining whether this is a BigVAR bug, a frequencyConnectedness bug, or a case where the user should build their own structures to simulate what cv.BigVAR would normally be producing.

extrospective commented 3 years ago

Amusingly, (just found) the assertion check code will fail if the second parameter to gran is 1. This can occur when passing in multiple own lambdas where the second happens to be 1.

Example:

Model1 <- constructModel(mdata, p=2, "OwnOther",ownlambdas=TRUE, gran=c(2, 1, 3), intercept=FALSE)
Model1Results = cv.BigVAR(Model1)

The fix suggested above will rectify this issue as well.

So for those users who want to try an own lambda=1, there is also a temporary need to workaround by not passing as the second lambda to gran.

extrospective commented 3 years ago

For reference, here is the code from frequencyConnectedness which fails when BigVAR.est is used rather than cv.BigVAR:

irf <- function(est, n.ahead, ...) {
    if (class(est) %in% c("varest", "vec2var")) {
        # cat("The model is from the vars package, using irf function from there.")
        return(vars::irf(est, n.ahead = n.ahead, boot = F, ortho = F))
    } else if (class(est)=="BigVAR.results") {
        # cat("The model is from BigVAR package, using own irf function.")
        return(irf.bigvar(est, n.ahead = n.ahead))
    } else {
        stop("Unsupported class of estimate")
    }
}

These are the first lines of the R/fevd.R file.

When using BigVar.est() results, the code drops to the else clause with the Unsupported error.

wbnicholson commented 3 years ago

When I invoke with a single parameter for gran, this code is intended to tell me I have one penalty parameter.

This has been corrected (6512e83) to only check if user-provided lambdas are included.

I want to show that BigVAR and VAR are equivalent at lambda=0, before starting to vary lambda. This is not directly possible since lambda=0 is explicitly allowed (not sure why) and I can approximate with lambda = 0.00000001. FYI. I think it would be nice to support lambda=0. Is there a reason why lambda=0 cannot be directly supported?

It's generally not advisable to run a penalized regression with lambda set to zero as doing so renders the penalty structure meaningless. If you want a baseline VAR estimate, it's much more computationally efficient to just run a least squares model. Additionally, some of the algorithms break down or have accuracy issues with lambda values close to zero (the optimizer tolerance will be larger than the penalty). If you want a least squares baseline, the package includes the function VARXFit, which if ic is set to NULL will fit a least squares VARX with the specified lag orders

library(BigVAR)
data(Y)
Bhat <- VARXFit(Y,p=4, IC=NULL)$Bhat

But when I use BigVAR.est (to test lambda=0.0000001), I encounter problems calculating FEVD with frequencyConnectedness::genFEVD. The latter is designed to work with BigVAR, but explicitly checks the class returned from the fit function (cv or est). cv.BigVAR returns a class which checks out with frequencyConnectedness. But BigVAR.est returns a (fairly limited) list type, and frequencyConnectedness refuses to report the FEVD since the class type assertion fails.

I was not familiar with this package. It apparently uses objects from the BigVAR.results class to construct the FEVD and irf. Consequently, I have changed the error when fitting with one parameter to a warning.