multiprocessing for seqmagick info

[cmccoy]

Hey Chris,

• Would you mind rebasing this against master? Some of these commits were already been included [e.g. in #32]
• I'd generally expect info to be I/O bound - does using multiple threads increase performance for you?
• Could multiprocessing be optional, disabled by default?

[crosenth]

• done
• yes
• done

[cmccoy]

This breaks SIGINT for me - the process hangs on ctrl-C.

  ➤ /home/cmccoy/development/seqmagick/seqmagick.py info 1.TCA.454Reads.fna
name               alignment    min_len   max_len   avg_len  num_seqs
^CCanceled.
^CCanceled.
^CCanceled.
^CCanceled.
^\[1]    5845 quit (core dumped)  /home/cmccoy/development/seqmagick/seqmagick.py info 1.TCA.454Reads.fnagg

[crosenth]

Unfortunately that is the nature of the multiprocessing module, http://docs.python.org/2/library/multiprocessing.html#pipes-and-queues, ctrl-c will kill child processes while the parent will hang waiting for the (terminated) children.

Decided to split this off into a new branch (multiprocessing) and reset the head to eb5a85a9d.

[cmccoy]

Yes - I was thinking that the default could be the old method, and only use multiprocessing at all when threads > 1, like:

        ssf = partial(summarize_sequence_file, file_type=arguments.input_format)
        if arguments.threads > 1:
            pool = multiprocessing.Pool(processes=arguments.threads)
            rows = pool.imap(ssf, arguments.source_files)
        else:
            rows = (ssf(f) for f in arguments.source_files)

[crosenth]

Done. ^^ merged with master/multiprocessing