search

TheFeshy / BookSift

A plugin for Calibre to do book identification and duplicate detection using the book's text instead of metadata
2 stars 0 forks source link

Multithreading #6

Open TheFeshy opened 12 years ago

TheFeshy commented 12 years ago

Implement multi-threading due to the parallel nature of our tasks

TheFeshy commented 12 years ago

Unfortunately, Python's multiprocessing module forces a tradeoff of either very simple shared objects, or huge overhead for complex shared objects. Due to the data-intensive nature of book sorting, this overhead would eat all of our performance gains and then some. Simple objects require complex management; therefore we will avoid them as well for now. Additionally, even simple objects are not handled properly under windows; this would require our entire data store to be duplicated per process!

That leaves us with python threading, which will give us performance gains for non-cpu bound tasks only. Hopefully this will still give us a performance improvement.