PHP Fatal error - memory limit

[bdolor]

My PHP memory limit is set to 256MB, the action taken to generate the error was to click on Lists > TOC.

It appears iterating through large books requires more than that which is problematic. Full stack trace below:

PHP Fatal error: Allowed memory size of 268435456 bytes exhausted at /private/tmp/php54-6h1C/php-5.4.30/Zend/zend_hash.c:253 (tried to allocate 82 bytes) in /Users/bpayne/Sites/pressbooks/wp-includes/post.php on line 671 [09-Sep-2014 22:14:00 UTC] PHP Stack trace: [09-Sep-2014 22:14:00 UTC] PHP 1. {main}() /Users/bpayne/Sites/pressbooks/wp-admin/admin.php:0 [09-Sep-2014 22:14:00 UTC] PHP 2. do_action() /Users/bpayne/Sites/pressbooks/wp-admin/admin.php:212 [09-Sep-2014 22:14:00 UTC] PHP 3. call_user_func_array() /Users/bpayne/Sites/pressbooks/wp-includes/plugin.php:505 [09-Sep-2014 22:14:00 UTC] PHP 4. PressBooks\Admin\Laf\display_list_img() /Users/bpayne/Sites/pressbooks/wp-includes/plugin.php:505 [09-Sep-2014 22:14:00 UTC] PHP 5. PressBooks\Lists\Lists_List_Table->prepare_items() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/admin/pb-admin-laf.php:909 [09-Sep-2014 22:14:00 UTC] PHP 6. PressBooks\Lists\Lists_List_Table->getItemsData() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/modules/lists/class-pb-lists-list-table.php:327 [09-Sep-2014 22:14:00 UTC] PHP 7. PressBooks\Lists\XpathList->getFlatArray() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/modules/lists/class-pb-lists-list-table.php:523 [09-Sep-2014 22:14:00 UTC] PHP 8. PressBooks\Lists\ListChapter->getFlatArray() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/modules/lists/class-pb-xpathlist.php:189 [09-Sep-2014 22:14:00 UTC] PHP 9. PressBooks\Lists\ListNode->getNodeAsArray() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/modules/lists/class-pb-listchapter.php:120 [09-Sep-2014 22:14:00 UTC] PHP 10. PressBooks\Lists\XpathList->getOnGoingNumberOfChild() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/modules/lists/class-pb-listnode.php:88 [09-Sep-2014 22:14:00 UTC] PHP 11. PressBooks\Lists\ListChapter->getOnGoingNumberOfChild() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/modules/lists/class-pb-xpathlist.php:231 [09-Sep-2014 22:14:00 UTC] PHP 12. pb_get_section_type() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/modules/lists/class-pb-listchapter.php:95 [09-Sep-2014 22:14:00 UTC] PHP 13. PressBooks\Taxonomy\chapter_type() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/functions.php:274 [09-Sep-2014 22:14:00 UTC] PHP 14. get_the_terms() /Users/bpayne/Sites/pressbooks/wp-content/plugins/pressbooks/includes/pb-taxonomy.php:253 [09-Sep-2014 22:14:00 UTC] PHP 15. get_post() /Users/bpayne/Sites/pressbooks/wp-includes/category-template.php:1247 [09-Sep-2014 22:14:00 UTC] PHP 16. WP_Post::get_instance() /Users/bpayne/Sites/pressbooks/wp-includes/post.php:433 [09-Sep-2014 22:14:00 UTC] PHP 17. WP_Post->__construct() /Users/bpayne/Sites/pressbooks/wp-includes/post.php:662 [09-Sep-2014 22:14:00 UTC] PHP 18. get_object_vars() /Users/bpayne/Sites/pressbooks/wp-includes/post.php:671

[lukaiser]

Can you give me your book? So I can test it? I don't have such a big book.

[bdolor]

While I haven't tested any of these books to see if they will re-produce this fatal error, there are a few big books in our collection: http://opentextbc.ca/ This one is a pretty big one: http://opentextbc.ca/strategicmanagement/ Grab the WXR file, then import.

[lukaiser]

I can't reproduce the error. The strategic management book has a peak memory usage of 7.25 MB. The book is 67 chapters long with a TOC of 447 items. I think that is okay. Can you give me your php version and settings and the book? There must be a leak. Or is it even bigger?

[bdolor]

Sorry I haven't looked at this right away — had to get some things out of the way. It is possible that my PHP accelerator was leaking. I've removed APC and will test again this week.

[bdolor]

The fatal error appears to be related to APC, my bad. However, the same action (clicking on Lists > TOC) is still problematic for me: Loading the page pins the httpd process on my CPU to around 100% for about 3 minutes while the page is being generated. After about 3 minutes, some content displays but the httpd process stays pinned for another 3 minutes or so, and then without further action on my part, 'Parts' display on the page and the httpd process goes back to a decent level. I will try and track down a publicly available book where this behaviour can be reproduced.

[lukaiser]

Hmmm, I can't reproduce that either. I am around 2 seconds with the strategic management book. And I have no idea what it could be. I created a test patch: https://dl.dropboxusercontent.com/u/244540/Memory_usage_Test_.patch

It generates an output like this (example from the strategic management book):

Time: 0.39819900 1411973246 | Memory Usage: 3.75 MB | Info: Initial Memory Usage
Time: 0.39851200 1411973246 | Memory Usage: 3.75 MB | Info: Columns created
Time: 0.93855500 1411973247 | Memory Usage: 7.25 MB | Info: Data recived
Time: 0.93863300 1411973247 | Memory Usage: 7.25 MB | Info: Ordered
Time: 0.93869900 1411973247 | Memory Usage: 7.25 MB | Info: Pagination
Time: 0.93898600 1411973247 | Memory Usage: 7.25 MB | Info: Header Output
Time: 0.11280700 1411973248 | Memory Usage: 7.25 MB | Info: List displayed
Time: 0.11284200 1411973248 | Memory Usage: 7.25 MB | Info: Memory Usage at the End

Peak of memory usage: 7.25 MB

Number of Chapters: 77
Number of Characters: 961256
Maximal Number of Characters in a Chapter: 37731
Number of Headings: 370
Number of Images: 189
Number of Tables: 4

[bdolor]

Here's a link to a very exciting screen capture of the behaviour described above. It is about 9 minutes long and the 'exciting' bits happen at 5 minutes 22 seconds (when the page loads) and then again at 8 minutes 54 seconds when the parts load and the CPU process spins down.

http://share.bradpayne.ca/public.php?service=files&t=d3e18950b64bbeff7ad30855a1b41c6d

Here are the stats on the above book with the patch file you've provided:

You can recreate similar behaviour, though likely not as extreme as the one above if you import a couple/few big books into one book. I did this with Introduction to Sociology (not yet public) and Introduction to Chemistry and saw a 20 second page load time. Here are the stats below from that amalgamation of Chemistry and Sociology, using the patch file you've provided:

[lukaiser]

Ou, we are talking books of that dimension? I thought the one you gave me was big ;) I try to find the time consuming part.

[lukaiser]

Okey. I came down from 13 to 2 seconds:

Time: 0.01977400 1412081687 | Memory Usage: 3.75 MB | Info: Initial Memory Usage
Time: 0.02028500 1412081687 | Memory Usage: 3.75 MB | Info: Columns created
Time: 0.15303300 1412081691 | Memory Usage: 13.75 MB | Info: Recived Lists
Time: 0.89600300 1412081698 | Memory Usage: 16.00 MB | Info: Array with Items
Time: 0.91707700 1412081698 | Memory Usage: 17.50 MB | Info: Data recived
Time: 0.91713700 1412081698 | Memory Usage: 17.50 MB | Info: Ordered
Time: 0.91727500 1412081698 | Memory Usage: 17.50 MB | Info: Pagination
Time: 0.91788100 1412081698 | Memory Usage: 17.50 MB | Info: Header Output
Time: 0.64657100 1412081700 | Memory Usage: 17.50 MB | Info: List displayed
Time: 0.64661300 1412081700 | Memory Usage: 17.50 MB | Info: Memory Usage at the End

Peak of memory usage: 17.50 MB

Time: 0.42602000 1412085003 | Memory Usage: 3.75 MB | Info: Initial Memory Usage
Time: 0.42634600 1412085003 | Memory Usage: 3.75 MB | Info: Columns created
Time: 0.67094400 1412085004 | Memory Usage: 15.25 MB | Info: Recived Lists
Time: 0.70443000 1412085004 | Memory Usage: 16.75 MB | Info: Array with Items
Time: 0.71413200 1412085004 | Memory Usage: 18.25 MB | Info: Data recived
Time: 0.71417200 1412085004 | Memory Usage: 18.25 MB | Info: Ordered
Time: 0.71423100 1412085004 | Memory Usage: 18.25 MB | Info: Pagination
Time: 0.71576100 1412085004 | Memory Usage: 17.50 MB | Info: Header Output
Time: 0.48087300 1412085005 | Memory Usage: 17.50 MB | Info: List displayed
Time: 0.48090800 1412085005 | Memory Usage: 17.50 MB | Info: Memory Usage at the End

Peak of memory usage: 18.25 MB

Number of Chapters: 128
Number of Characters: 6361456
Maximal Number of Characters in a Chapter: 108004
Number of Headings: 1370
Number of Images: 789
Number of Tables: 154

It is not really working yet with a lot of debugging stuff in it. So I prefer to not commit. But I would love to have feedback if I am any where near to a working version with a book as big as yours (After importing 60 chapters in batches of 3 I had enough). You find a zip here: https://dl.dropboxusercontent.com/u/244540/pressbooks-lists.zip

Thank you very much!

PS: I just read that wp_cache_set is not session persistent. If I don't get closer to some thing working, I guess that this is some thing I have to look at. Either a php cash or I generate and manage a mysql table. I love mysql views but it is not so common yet. What do you guys think about it?

[lukaiser]

Time: 0.35028100 1412156095 | Memory Usage: 3.75 MB | Info: Initial Memory Usage
Time: 0.35060100 1412156095 | Memory Usage: 3.75 MB | Info: Columns created
Time: 0.38826000 1412156097 | Memory Usage: 26.50 MB | Info: Recived Lists
Time: 0.62508000 1412156097 | Memory Usage: 31.75 MB | Info: Array with Items
Time: 0.69725200 1412156097 | Memory Usage: 36.25 MB | Info: Data recived
Time: 0.69731600 1412156097 | Memory Usage: 36.25 MB | Info: Ordered
Time: 0.69740100 1412156097 | Memory Usage: 36.25 MB | Info: Pagination
Time: 0.70499900 1412156097 | Memory Usage: 32.50 MB | Info: Header Output
Time: 0.35325200 1412156098 | Memory Usage: 32.00 MB | Info: List displayed
Time: 0.35328600 1412156098 | Memory Usage: 32.00 MB | Info: Memory Usage at the End

Peak of memory usage: 36.25 MB

Number of Chapters: 285
Number of Characters: 7482571
Maximal Number of Characters in a Chapter: 108004
Number of Headings: 4470
Number of Images: 2494
Number of Tables: 619

3 seconds are ok for a book of that size. Right? It feels natural to me and I try to change the ajax stuff to react instantly. That should allow a good workflow.

[bdolor]

remarkable speed improvement.

[lukaiser]

• 291f4bd33982eb9bdc8b8cd28442d5555afa23f6 improves the speed.
• d0a18dfc52dc5acd9aa4d8022d150e63624916e2 lets javascript handle the recalculation of the numbers in the list views. It makes it way smoother to work with list if a book is really big.