Add multi-threading / multi-processing

openpreserve / fido

Format Identification for Digital Objects (FIDO) is a Python command-line tool to identify the file formats of digital objects. It is designed for simple integration into automated work-flows.

http://openpreservation.org/technology/products/fido/

Other

147 stars 47 forks source link

Add multi-threading / multi-processing #9

Open adamfarquhar opened 14 years ago

adamfarquhar commented 14 years ago

The 0.5.x implementation appears to be IO bound. Throughput would be increased by moving file-reads to a separate thread so that they will happen in parallel with pattern matching.

One approach: add multiple workers, each of which reads, matches. Another: add a pool to do reads, and another to do matches.

But - it's all fast enough for now!

carlwilson commented 4 years ago

Delayed until API implemented, delayed until 1.7+

carlwilson commented 1 year ago

Hackathon 2023 review. Not selected for initial prioritisation. It could be best done when other issues are fixed, as it may be a trickier merge. Worth considering if this isn't just as well done via an invocation script