Open fbuchinger opened 8 years ago
@fbuchinger Thanks for your suggestion. Multiprocessing is only useful for CPU-bound computations. Reading EXIF data is almost certainly I/O-bound.
You have already noticed that a speed up of 18 is unlikely to result from using three CPUs instead of one, and indeed using three CPUs isn't the reason it's faster: Your chunk_list()
function is documented correctly as returning chunks of length n
, so fl_chunks
contains chunks of length 3. The multiprocessing version only operates on the first 9 files, so it's hardly a surprise it's faster. (No idea how many files your test directory contains.)
A proper benchmark would also have to take caching into account. When you have already read the metadata of the same files before, chances are that the contents of the directory is already cached by the operating system. In this case, reading the files will be very fast, since no disk access is required, so you are more likely to be able to make use of more than one CPU. In real applications, it's more likely that the image files haven't been pulled into the cache yet. So for a proper benchmark, you would have to make sure that your test files haven't been cached yet.
@smarnach: thanks for your hints! I've updated the multiprocessing part of my script and I am now using multiprocessing.Pool
and its map method to do the parallelisation. Unfortunately the multiprocessing version now takes even slightly longer than the single-executable version:
Exiftool internal batch took 10.4846533613 sec
Exiftool Stay Open/External batch took 11.6878449432 sec
Exiftool multiprocessing batch took 11.5929283865 sec
I've created a gist of my script https://gist.github.com/fbuchinger/7b8d68e967e8a0600b79. Just out of interest: does the multiprocessing part look ok or is there another mistake in it? I've monitored the Windows task manager and just found one copy of exiftool actively working (i.e. increasing/decreasing its amount of consumed memory)
@fbuchinger I had a quick look. Since you use / 3
as the chunk size, this might leave a few single items at the end of the list due to rounding that require another exiftool run, but the effect of that is likely to be minor.
I don't know much about Windows. Generally I'd look at CPU usage to decide whether a process is doing anything, but since what exiftool does is mostly I/O, this might be misleading in this particular case.
Currently pyexiftool leverages the
stay_open
feature of exiftool to get a speed up compared to invoking one exiftool instance for each image. What if we took the ball further and used the python multiprocessing module to invoke multiple exiftool stay_open instances and split the load between them? Here is a script that compares the "internal" batch mode of exiftool, the "external" batchmode using pyexiftool and the "multiprocessed external" batchmode using multiprocessing:When I run this script on the Canon sample images of Phil Harvey's metadata repository, I get these results on my machine:
Exiftool internal batch took 14.2978597714 sec Exiftool Stay Open/External batch took 12.7552563562 sec Exiftool multiprocessing batch took 0.714086082069 sec
I have to admit 0,71 for multiprocessing sounds to good to be true, so there might be some error in the script. nevertheless, a previous version with two workers took 7,77 seconds, so there is quite a performance gain possible.
Do you plan to add multiprocessing functionality to pyexiftool?