Large k, small T scenario

[sentian]

Hey Will,

Thanks for the great R package. I came across a case where my data is a panel data with many cross-sections and relatively short time period (k in thousands and T in 30s). Typical VAR won't work due to the k>T issue. Since LASSO can handle high dimensions, I hope VAR-LASSO is a possible solution.

So I give a shot at BigVAR. It seems to be working fine (used the basic LASSO) and provides me some estimates, but it comes with a warning of k being bigger than T (from the function 'constructModel'). I'm wondering whether this scenario has been justified before. I did not find evidence in your paper. So I'm hoping to gain some insights from you.

Thanks.

[wbnicholson]

This scenario is feasible under the specifications of BigVAR, however, it does put a bit of a strain on the efficacy of rolling cross validation, which, by default, splits the data into thirds to perform penalty parameter selection and evaluation of forecast accuracy. In addition, in order to construct the lag matrix, we need to use p observations for initialization. In your setting, this would leave you with very few observations in which to make a judgement on the penalty parameter.

My recommendation would be to try with very small p (1 or 2) and to use the "leave one out" approach (cv="LOO") as opposed to the default rolling cross validation.