Time-out when XML export includes images and videos

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start an all-models-all-image-all-video XML export.
2. Wait and see it takes more than an hour (3600 secs) to complete.
3. Get a fatal 'Maximum execution time exceeded' error.

What is the expected output? What do you see instead?
One would want a more rapid XML-export that doesn't take more than, say 10 
minutes to complete a complete export.

Original issue reported on code.google.com by fwp...@gmail.com on 1 Jul 2012 at 8:27

[GoogleCodeExporter]

Tried a complete export today, on my 2.6GHz-4GB, Ubuntu Precise system. I 
started it from the commandline using

$ php5 -f ./download_xml.php includeimages=true includevideos=true | tee 
~/CandyDollDB_2012-07-01.xml

It ran for one hour, then got killed by PHP:

PHP Fatal error:  Maximum execution time of 3600 seconds exceeded in 
/var/www/candydolldb/class/class.tag2all.php on line 29
PHP Stack trace:
PHP   1. {main}() /var/www/candydolldb/download_xml.php:0
PHP   2. XmlOutputModel() /var/www/candydolldb/download_xml.php:193
PHP   3. Tag2All::FilterTag2Alls() /var/www/candydolldb/download_xml.php:101
PHP   4. Tag2All->getModelID() /var/www/candydolldb/class/class.tag2all.php:190

Since watching the scripts' output, I am convinced the most time is spent in 
Tag-filtering and Date-filtering. I would like to see a more efficient filter 
mechanism. If that means calling an extra DB-query, so be it.

Original comment by fwp...@gmail.com on 1 Jul 2012 at 8:34

[GoogleCodeExporter]

Attached is a screenshot proving my point. It shows the vast majority of 
processing (88%) takes place in Tag2All::FilterTag2Alls.

This analysis was created with the XDebug profiling output of a full XML-export 
of VikaZ and loading this into KCacheGrind.

Original comment by fwp...@gmail.com on 1 Jul 2012 at 6:11

Attachments:

• Screenshot_KCacheGrind_XML-export_VikaZ.png

[GoogleCodeExporter]

Yeah 1 hour does seem a little long

Original comment by mranimos...@gmail.com on 2 Jul 2012 at 8:52

[GoogleCodeExporter]

Tweaked the filtering of tags in r528. Instead of filtering ALL tag2alls each 
and every time, it now filters the big pile for all a model's tags, with the 
other parameters set to FALSE. It then filters this array during set-, image- 
and video-tagging.

It seems a LOT more efficient. The full export run of ALL models now takes less 
than 8 minutes. Issue resolved, I would say.

Original comment by fwp...@gmail.com on 3 Jul 2012 at 4:03

• Changed state: Fixed