Vector Memory Exhausted in R-lean

[michaeltopper1]

Hi @s3alfisc,

I'm having trouble getting the boottest function to work with my data set. My best guess is that this is a memory problem given the large amounts of observations I have (~4 million) and high dimensional fixed effects (time = 2536, group = 22). Plus, I keep getting same error: Error: vector memory exhausted (limit reached?)

I have tried using the argument engine = "R-lean" although I still get the vector memory exhausted error.

Moreover, I also tried out the summclust package, but it looks like I'm running into a similar problem there.

I would put in a reprex, although I don't know how helpful that would be here.

Is there a solution around this problem? My only thought is to run this on a server somewhere, but I wanted to stop by here in case there is an easier way.

[s3alfisc]

Hi @michaeltopper1 , thanks for reporting!

Some quick questions:

• How large is your number of observations? -> N = 4Mio
• How large the number of covariates (including all fixed effects minus the one you project out via the fe argument)? 2000 + 20 + any other covariates
• How many bootstrap iterations do you run?
• How many clusters do you have? 22?

Actually, I think this should be doable?

R-lean should help if your memory error is caused by a bootstrap weights matrix that gets too large - the weights matrix is of dimension G x (B+1). The advantage of the R-lean algo is that it does not allocate the entire weights matrix prior to the bootstrap, but only afterwards.

Ad hoc, I have two ideas what you could try:

• try the fast and reliable algos - they are implemented as sparse matrices in the main branch (but not yet on CRAN), which might save you some memory. The 11 algo is equivalent to what you should be getting from the fnw11 algo, just less vectorized. This only helps if it is not a large weights matrix that causes the memory error (which I don't think is the case, I recall another error message).
• try WildBootTests.jl, which is far more memory efficient than the fnw11 bootstrap in fwildclusterboot. I have at some point written this tutorial to help install Julia and WildBootTests.jl, maybe it is helpful?

I will be super busy tomorrow, so cannot promise I'll be able to help, but will try to take a closer look on Wednesday & respond here =)

Best, Alex

[s3alfisc]

Also, in case you are not already doing it, you should definitely set fe = time, which should help a lot?

[s3alfisc]

Sorry, of course I'd summclust also fails with the same error, the it cannot be the weights 😅

[michaeltopper1]

Thanks for the quick response!

To answer the Q's:

• How large is your number of observations? -> 3,582,560
• How large the number of covariates (including all fixed effects minus the one you project out via the fe argument)? 2535 + 21
• How many bootstrap iterations do you run? 999, although using 101 also has the same issue
• How many clusters do you have? 22

Here's an output of my regression. Sorry for the corny screen shot,

And the corresponding fwildclusterboot code:

boottest(entry_d, clustid = c("district"), B = 999 , param = "treatment", engine = "R-lean")

I'm guessing boottest sticks all of these matrices into RAM? If that's the case, then I think I'll be able to solve this by getting on a server.

[michaeltopper1]

Ok your final comment on setting fe = "date" helped A TON. It now runs fast. Sorry about that! I should've read a little more into the documentation on this note. I'll close this now.

Thanks for the help!

[s3alfisc]

Awesome, great that you could get your bootstrap to run! No need to apologize, I am always super happy to hear from users of my packages, and to see that they are used in actual research projects! =)