search

peircej / jbrout

Automatically exported from code.google.com/p/jbrout
0 stars 0 forks source link

Slow operations on large picture sets (10K+) #144

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. Add (or refresh) a folder with many photos (10K+)
2. You have time for break ;)

What is the expected output? What do you see instead?
I can see that the import process takes ages, and it looks like the
progress bar is refreshed after each photo import.
I would expect the import to be quicker, and I wouldn't mind if the
progress bar was refreshed less often.

What version of the product are you using? On what operating system?
svn revision 282 on Ubuntu 9.10

Please provide any additional information below.
I have investigated the performance issue with a dummy loop: no other
action than calling showProgress() at every step.
For 20000 (dummy) steps, the process is way too long (46 seconds).
It seems that most of the time is wasted refreshing the user interface.
Changing showProgress() to sub-sample the user interface refresh allows to
execute the whole dummy loop (as expected) in less than one second!
--> Therefore I recommend sub-sampling the user interface refresh in the
showProgress() function.
Please find the details attached in .txt file. (patch 282 in .diff files)

Wonderful product, by the way ;)

Original issue reported on code.google.com by julien.d...@gmail.com on 23 Feb 2010 at 10:10

Attachments:

GoogleCodeExporter commented 8 years ago 
Great, I will apply your patch to night ...

But have you made some real tests, in context ? Do you feel it really better ?

Original comment by manat...@gmail.com on 23 Feb 2010 at 10:16

GoogleCodeExporter commented 8 years ago 
You're right, that was only theory... ;)

In fact, for 2300 pictures, the import duration is reduced from 39 to 32 
seconds,
which is not absolutely fantastic, I must admit.
For 11603 pictures: 158 vs 202 seconds, once again ~20% better.
In the meanwhile the display is more quiet, hence my better feeling.

I definitely recommend testing and feeling it! (also try different r values)

Original comment by julien.d...@gmail.com on 23 Feb 2010 at 12:14

GoogleCodeExporter commented 8 years ago 
By the way, here is another (very simple!) patch to avoid useless
PhotoCmd.__refresh() calls during the "auto" transform process when 
sens=="none".
For 1160 pictures, the album refresh timings are: 
26.9/22.5/17.2 (original/showProgress/showProgress+autotransform)
This is nearly one third faster, after all...

Original comment by julien.d...@gmail.com on 23 Feb 2010 at 2:06

Attachments:

GoogleCodeExporter commented 8 years ago 
And finally, the last patch, reducing the xpath() calls in db.__add().
For 1164 pictures we are now at 12 seconds, as compared to 26 (x2 speed)
For 11608 pictures, we are now at 82 seconds, as compared to 216 (x3 speed)

I also include the global patch for all three changes from revision 282.

Do not hesitate to comment or signal anything strange in my changes.
Optimizing is so funny... ;)

Original comment by julien.d...@gmail.com on 24 Feb 2010 at 12:40

Attachments:

GoogleCodeExporter commented 8 years ago 
Patches applied in r296, r297 and r298

Original comment by r...@wallace.gen.nz on 3 Jun 2010 at 9:00