Addition of parallelization to multiple procedures & user-defined concatenation order

[alephnull7]

Changes that specifically address listed issues

• Extraction of GenBank files has been parallelized. Due to the nature of the operations done, multiprocessing was determined to be the best approach. This did require some refactoring, which can be noted in changes to the PlastidData class, the creation of the PlastidDict class, and the creation of helper methods related to the extraction of a sublist of files as well as individual files. From these new helpers, you will also see that multiprocessing occurs on sublists of the overall file list, as the overhead related to extracting each file in a separate process is less performant than single processing.
• The new user-provided argument --order is available, which relates to the concatenation order of the alignments. By default, the ordering is seq (sequence), corresponding to the sequence ordering of the features in the first input genome. With parallelization of extraction now occurring, for the seq option, the first file is extracted before the rest are parallelized to guarantee the specified order. The other option is alpha (alphabetic) which corresponds to the features being concatenated in alphabetic order.

Other changes

• The collection of successful alignments has also been parallelized by using multiprocessing. This choice was made for similar reasons as extraction and also uses sublists for the processes. As the collection of each nucleotide alignment does take much longer than the file extraction, the use of sublists is not as important, but there are minor gains compared to parallelizing the alignment collection individually.
• The use of the mafft wrapper provided by Biopython has been replaced by a subprocess call to mafft. There is no performance difference related to this update and was changed to prevent the depreciation of Application from causing issues in the future. The parallelization of mafft was investigated for improvements, but no obvious changes could be found. There may be changes to the parallelization of mafft and other processes that would decrease execution time, but that would probably involve targeting the software for a specific machine configuration.
• Both of the final concatenated alignment files are now written concurrently. This was mostly accomplished by taking advantage of the export_fasta method available to alignm_concat so that the Nexus file does not have to exist before the FASTA one.

[michaelgruenstaeudl]

The use of the mafft wrapper provided by Biopython has been replaced by a subprocess call to mafft. There is no performance difference related to this update and was changed to prevent the depreciation of Application from causing issues in the future.

Yes, that is very sensible! While maintained by a small team of enthusiastic volunteers, Biopython does not have a large developer base. It is a good idea to wrap mafft via a regular subprocess call.

[michaelgruenstaeudl]

Extraction of GenBank files has been parallelized. Due to the nature of the operations done, multiprocessing was determined to be the best approach. This did require some refactoring, which can be noted in changes to the PlastidData class, the creation of the PlastidDict class, and the creation of helper methods related to the extraction of a sublist of files as well as individual files. From these new helpers, you will also see that multiprocessing occurs on sublists of the overall file list, as the overhead related to extracting each file in a separate process is less performant than single processing.

I am not sufficiently familiar with the different types of process parallelization to judge which one is the best option. I fully trust your choice here.