Parallelize source file processing

[noteflakes]

When processing a directory, things can be parallelized:

• Parsing individual files can be performed by a thread pool. Generating an AST and event stream for each file can be completely parallelized. The actual generation of lilypond code can be serialized using a critical section or some other lock.
• This actually should be relatively simple:
  • Work#process_directory starts a worker thread pool and adds each encountered file to the worker queue.
  • Workers pull file names from the worker queue and process them using process_lydown_file.
  • Work#process_lydown_file uses a critical section or other lock to synchronize the last stage of processing, the actual translation to lilypond code is serialized.

[noteflakes]

Hmm, this seems to be a bit more complicated than I thought. An alternative approach:

• Process directory and create a tree structure of movements and their files.
• Put all files to process into array, use Parallel to read files and convert to stream.
• Iterate over tree and call #process with corresponding streams, inside of #preserve_context calls.

[noteflakes]

OK, parsing in parallel shaves off a few seconds, but still work to do on translating to lilypond. For that to happen we will need to turn the translate function into a pure function that gives a bunch of lilypond streams as result, without affecting the work state. In general, #preserve_state is a bit hacky, if we need to use it each time we process a file.

[noteflakes]

For more performance improvements, see #43

[noteflakes]

Another possibility of parallelizing compilation is to take advantage of movement page breaks, which when rendered create separate \bookPart sections. Each section is rendered separately, then compiled separately. The result is then combined into a single PDF using pdftk.

[noteflakes]

Finally we don't use pdftk for PDF merging (another tool to install, ruby gem has dependencies the size of the Himalayas), but rather a pure ruby gem called combine_pdf.

[noteflakes]

Not so fast, Dark Helmet! Silly me, I forgot all about page numbers. When dividing into bookparts, each bookpart will start at page 1. So , for now we disable parallelized bookparts, until we find an acceptable solution. See commit ce3866e.