Open PhDyellow opened 1 year ago
Yes, it does constrain memory usage.
However, it adds almost a whole second to each batch.
20 batches of 100 rows took 30s with two gc()
's, 15s with one gc()
, and 0.19s with no gc()
.
If all else fails, it might be ok, with much larger batches. It will almost certainly be the limiting factor.
See c9c2166