Async rendering causes too many open files and no caching

[bbor]

Hi, first of all thanks for this great module! In testing with large numbers of input files (~3500), I consistently get an EMFILE: too many open files error. It seems that metalsmith-layouts is asking consolidate to render all the input files asynchronously. Each of my input files is using the same template, so what should happen is that consolidate should read the template the first time, then store it in the cache so that it doesn't have to be opened and re-read for subsequent input files. But instead, because all the render jobs are started asynchronously at the same time, consolidate actually begins reading the template file immediately for each input file, before any of its readFile calls actually finishes. The result is that once the number of input files grows large enough, boom, too many open files. By changing the each() call to a synchronous forEach() loop I can make the problem disappear, but I'm sure there must be a better and more generalizable solution. Thoughts?

[ismay]

Ah yeah. We could replace each with eachLimit (https://caolan.github.io/async/docs.html#eachLimit). Set the default to a sensible amount and allow the user to override it. That should take care of it. Feel like submitting a PR?

[bbor]

Yeah, eachLimit would cut the batch size down, but it doesn't really solve the problem. Within a batch of input files, each time we come across a request for a template that hasn't been cached from a previous batch, we'd still open that template file as many times as we have input files in the batch. Right? I think we would need to first save a list of all the templates that our input files will be using, then ask consolidate to do a render with each of those templates so that it caches the template contents. After all those initial pre-loads are done, then we can do the async each() call to loop through all the input files.

[bbor]

submitted a PR for discussion. Cheers!

[ismay]

Closed from #120