Throttle oversized reports from reporter.php

[ompiteam]

I encountered a problem with generating a colossal-sized report. For example, the following inputs to {{{reporter.php}}} (which attempt to show every test case submitted to the database over the past two weeks) will generate a blank screen to the user:

{{{ Timestamp: Past Two Weeks Phase: Test Run }}}

Also, the following message turns up in the httpd {{{error_log}}}.

{{{ PHP Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 32 bytes) in database.inc on line 340, referer: reporter.php }}}

{{{reporter.php}}} should be adjusted to throttle the boatload of incoming rows, e.g., ''"this report is too huge for the server, I will show it to you in chunks"''.

[ompiteam]

Imported from trac issue 62. Created by emallove on 2006-09-04T17:00:36, last modified: 2008-07-22T10:43:58

[ompiteam]

Trac comment by emallove on 2006-09-07 14:47:38:

(In [314]) * Added a {{{$help}}} variable and {{{help()}}} function

• Added explanations for "By Test Case" and "Phase" labels above pass/fail columns
• Temporarily remove [i] links from {{{summary.php}}}. To be added back in after {{{dump_report()}}} (and/or reporter.php as a whole) is made more memory efficient (see #62). Basically, we either need to increase our php's memory_limit beyond 16MB, or not store the whole report's HTML string in one variable (it's a problem when a report is bigger than 4MB). There must be a good reason for setting the memory limit at 16MB, right? We could make the [i] links dynamic somehow, rather than pulling all the auxillary [i] link data over at once, as the [i] links at times put the report over the top of the limit.
• {{{$level_info}}} was stale HTML from {{{summary.php}}}
• sql_to_en should return an array, not an HTML string
• reporter.inc breaks down into "string processing" and "developer/user helper" functions

[ompiteam]

Trac comment by emallove on 2006-09-08 17:10:51:

(In [317]) * {{{reporter.php}}} was apparently being very wasteful with memory (see #62). Reason being was, in the refactoring process the functions {{{dump_screen()}}} and {{{dump_report()}}} were created which return html strings. The problem was that those strings were created in a global scope. Now said html string vars have been put in a function so they must be going out of scope and saving memory. (We still max out on a 2500-5000 row Test Run report. A workaround would be to prune the results using the menus and/or filters.) The reason for variablizing these particular html strings is that the code is more manageable for dealing with switches for custom reporters (e.g., summary.php) that use {{{reporter.php}}}. E.g.,

{{{ just_results - hide query description no_help - show inline help blurbs info_links - add some helpful pointers at the end of the report }}}

• Because the above was done, we can now put {{{[i]}}} link details back into {{{summary.php}}}.

• Factored dump_report() down to:

  • {{{filters_description($filters)}}}
  • {{{get_selects()}}}
  • {{{create_phase_level_query($phase)}}}
  • {{{get_phase_selection($params)}}}
  • {{{info_pointers()}}}

  ({{{dump_report()}}} is still seems a bit long (300 lines).

• Put copyrights in php comments (not html comments) so we do not send web/email clients repeatedly
• Simplified implicit and explicit phase selection
• {{{$config}}} was not needed for results filtering (why was it ever?)

[ompiteam]

Trac comment by jsquyres on 2006-09-12 14:06:28:

First take on this is to change database.inc:pgquery() to simply return the pg_query() result handle, and have the caller slurp the data directly from the result handle (vs. manually pushing each row into a new array and therefore taking twice the memory).

This will save a good chunk of memory, but we still may be in trouble if some queries just return so much data that pg_query() itself fails. Leaving this ticket open to see if that will continue to be a problem, or if we want to handle it better in milestone 2.0 (e.g., show X results at a time).

[ompiteam]

Trac comment by emallove on 2006-09-13 09:46:46:

(In [324]) * For memory savings, {{{pgquery}}} returns a {{{pgsql resource}}} instead of a 2D array (see #62)

• Validate {{{db}}} parameter
• Place postgres username and password in {{{dbpassword.inc}}} (outside of svn tree)
• Enlarge help font

[ompiteam]

Trac comment by jsquyres on 2006-09-13 14:17:05:

Let's put a max number of rows in the current code. That is, display the first N rows (say, N = 1000 or something) and at the bottom of the table, say, "Max number of rows exceeded; not all the data shown. Please limit the search more..." (or something like that).

This is better than the html output suddenly stopping because PHP killed the script after 30 seconds of execution time. That way the user knows that the search was too big and they need to limit it.

[ompiteam]

Trac comment by emallove on 2006-09-27 12:52:14:

The following report-size related message can not be addressed by {{{pg_num_rows}}} because {{{reporter.php}}} does not get the result back in time to count the rows/fields.

{{{ ... PHP Fatal error: Maximum execution time of 30 seconds exceeded in ... reporter.inc on line 138 }}}

These may address the above error (roughly in order of easiest to hardest):

• Disallow detailed reports to go back farther than 1 week
• If the server is not running PHP in ''safe mode'', increase the server's execution time_limit in the {{{php.ini}}} file. Otherwise, use {{{set_time_limit()}}}
• Database optimizations in MTT2.0 (long queries may execute faster than 30 seconds)
• A javascript keepalive (?) window of some kind to prevent the script from timing out

[ompiteam]

Trac comment by emallove on 2006-09-27 15:33:10:

(In [349]) closes #62

Crude oversized report throttling (this will be improved with #79).

[ompiteam]

Trac comment by emallove on 2007-09-05 16:38:11:

This old favorite has returned. Clicking the "Detail" button from http://www.open-mpi.org/mtt/index.php?do_redir=316, incurs this error:

{{{ $ tail -f /var/log/httpd/www.open-mpi.org/error_log

[client 192.18.128.5] PHP Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 41 bytes) in /nfs/rontok/xraid/data/osl/www/www.open-mpi.org/mtt/screen.inc on line 262, referer: http://www.open-mpi.org/mtt/index.php?limit=&wrap=&trial=&yaxis_scale=&xaxis_scale=&hide_subtitle=&remote_go=summary&do_cookies=1&phase=test_run&text_start_timestamp=2007-08-31+06%3A35%3A53+-+2007-08-31+18%3A35%3A53&text_platform_hardware=%5Ex86_64%24&show_platform_hardware=show&text_os_name=%5ELinux%24&show_os_name=show&text_mpi_name=%5Eompi-nightly-v1.2%24&show_mpi_name=show&text_mpi_version=%5E1.2.4b1r15976%24&show_mpi_version=show&text_suite_name=%5Eibm%24&show_suite_name=show&text_test_name=all&show_test_name=hide&text_np=%5E8%24&show_np=show&text_full_command=&show_full_command=show&text_http_username=%5Ecisco%24&show_http_username=show&text_local_username=all&show_local_username=hide&text_platform_name=%5Esvbu-mpi%24&show_platform_name=show&phase=test_run&test_result=_rp&text_os_version=&show_os_version=hide&text_platform_type=&show_platform_type=hide&text_hostname=&show_hostname=hide&text_compiler_name=&show_compiler_name=hide&text_compiler_version=&show_compiler_version=hide&text_vpath_mode=&show_vpath_mode=hide&text_endian=&show_endian=hide&text_bitness=&show_bitness=hide&text_configure_arguments=&text_exit_value=&show_exit_value=hide&text_exit_signal=&show_exit_signal=hide&text_duration=&show_duration=hide&text_client_serial=&show_client_serial=hide&text_result_message=&text_result_stdout=&text_result_stderr=&text_environment=&text_description=&text_launcher=&show_launcher=hide&text_resource_mgr=&show_resource_mgr=show&text_network=&show_network=show&text_parameters=&show_parameters=hide&text_trial=%5Ef%24&lastgo=summary }}}

[ompiteam]

Trac comment by jsquyres on 2007-09-17 11:57:35:

Ethan: given that you moved this to "critical", do you also want to move it back to v3.0?

[ompiteam]

Trac comment by jjhursey on 2008-02-05 10:41:05:

In the v3 reporter change I seemed to have reverted the fix in r613. The problem is that since we do not use temporary tables any more, we bring over the whole query result to the client. So if PHP is not able to handle this it just blanks out :(

A work around is to increase the memory limit of the reporter to something like 32 MB.

[ompiteam]

Trac comment by jjhursey on 2008-02-05 10:45:07:

Temporary work around in r1153.

In my opinion, this is not a blocker since most people will not try to view that many details at once. The typical use case is for drilling down to a small set of results and then viewing the details. This work around should allow for the majority of queries to the reporter to finish successfully. I wish we could print an error when we do exceed the limit so the user doesn't just see a blank screen.

[ompiteam]

Trac comment by emallove on 2008-07-22 10:43:24:

The Reporter is now barfing on the pg_fetch_all call in the get_form_universe function. This is very bad, because the only point of that function is to load the insertion pop-up window links, which are seldom used. The Reporter needs to check whether the number of rows returned in the pg_fetch_all command is too large, and in that case, punt on the insertion popup windows.

http://www.open-mpi.org/mtt/index.php?do_redir=747 (click the Test runs header link, then the Detail button)