heavy load average with php+cgi+suexec setup

crowell / modpagespeed_tmp

Automatically exported from code.google.com/p/modpagespeed

Apache License 2.0

0 stars 0 forks source link

heavy load average with php+cgi+suexec setup #204

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago

What steps will reproduce the problem?
1. Use mod_pagespeed with php+cgi+suexec
2.
3.

What is the expected output? What do you see instead?
A lot of errors, and heavy load 

What version of the product are you using (please check X-Mod-Pagespeed
header)?

0.9.15.3-r404

On what operating system?
Debian 6.0 (Squeeze)
Which version of Apache?
2.2.16-6
Which MPM?
prefork
Please provide any additional information below, especially a URL or an
HTML file that exhibits the problem.

I use php+cgi+suexec, this is a vhost example:

<VirtualHost XX.XXX.160.226:80>
ServerName XXX.it
ServerAlias *.XXX.it

LogFormat "%V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" 
combined2
UseCanonicalName Off
VirtualDocumentRoot /home/jail/home/XXX/httpdocs/%0/
Options Multiviews -Indexes +FollowSymLinks

SuexecUserGroup userXXX groupXXX
AddHandler php-fastcgi .php
AddType application/x-httpd-php .php
Action php-fastcgi /gcgi-bin/php-fcgi-starter
ScriptAlias /gcgi-bin/ /var/www/XXX-conf/gcgi-bin/
ScriptAlias /cgi-bin/ /var/www/XXX-conf/cgi-bin/

ErrorLog /var/log/apache2/host.XXX.it-error.log
CustomLog /var/log/apache2/host.XXXX.it-access.log combined2
</VirtualHost>

With this configuration I've heavy load average, and a lot of:

[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] Premature end of 
script headers: php-fcgi-starter
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] Premature end of 
script headers: php-fcgi-starter
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] <b>Security 
Alert!</b> The PHP CGI cannot be accessed directly.
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] 
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] <p>This PHP CGI 
binary was compiled with force-cgi-redirect enabled.  This
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] means that a page 
will only be served up if the REDIRECT_STATUS CGI variable is
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] set, e.g. via an 
Apache Action directive.</p>
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] <p>For more 
information as to <i>why</i> this behaviour exists, see the <a 
href="http://php.net/se
curity.cgi-bin">manual page for CGI security</a>.</p>
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] <p>For more 
information about changing this behaviour or re-enabling this webserver,
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] consult the 
installation file that came with this distribution, or visit 
[Tue Feb 01 18:48:58 2011] [error] [client XX.XXX.160.226] <a 
href="http://php.net/install.windows">the manual page</a>.</p>

Let me know if you need other info.

Giuseppe.

Original issue reported on code.google.com by g.iucul...@gmail.com on 1 Feb 2011 at 6:03

GoogleCodeExporter commented 9 years ago

It seems plausible that the load-average issues are related to the error 
messages you are seeing.  None of those are mod_pagespeed errors, but I presume 
you had a working configuration that stopped working when you turned on 
mod_pagespeed.  Is that right?

All of mod_pagespeed's error messages have the word 'mod_pagespeed' in them.  
But it's possible that something in the configuration is causing mod_pagespeed 
to issue a lot of warnings as well.  Could you try 'loglevel warn' or -- if you 
still see nothing -- 'loglevel info' and see if something related to 
mod_pagespeed shows up?

Failing this, it should be easy to start zeroing in on what causes the problem. 
 Start with using PassThrough mode only first, then add filters one at a time 
until the problem appears.

That will help us diagnose this better.

Original comment by jmara...@google.com on 1 Feb 2011 at 8:00

GoogleCodeExporter commented 9 years ago

Yes, that is right.

I got these warnings and errors:

[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/anime/shinrei-tantei-yakumo/small/shinrei-tantei-yak
umo-s31.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/anime/galaxy-angel/small/galaxy-angel-s13.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out: http://www.jigoku.it/images/manga/eater/small/eater-s0.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out: http://www.jigoku.it/forum/customavatars/avatar1039_35.gif
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out: http://www.jigoku.it/forum/images/smilies3/doh.gif
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/anime/kono-aozora-ni-yakusoku-wo/small/kono-aozora-n
i-yakusoku-wo-s1.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/anime/hime-chen-otogi-chikku-idol-lilpri/small/hime-
chen-otogi-chikku-idol-lilpri-s17.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/manga/3-three/small/3-three-s0.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out: http://www.jigoku.it/images/manga/katsu/small/katsu-s0.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/manga/appleseed/small/appleseed-s0.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/manga/bamboo-blade/small/bamboo-blade-s0.jpg
[Tue Feb 01 21:27:26 2011] [warn] [mod_pagespeed 0.9.15.3-404]
[0201/212726:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)]
Fetch timed out:
http://www.jigoku.it/images/anime/renketsu-houshiki-2nd/small/renketsu-houshiki-
2nd-s2.jpg
[Tue Feb 01 21:27:39 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/potemayo/small/potemayo-s11.jpg:0:
serf_context_run error status=110 (Connection timed out)
[Tue Feb 01 21:27:39 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/aika-r-16-virgin-mission/small/aika-r-16-virgi
n-mission-s7.jpg:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:27:39 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/bleach-the-movie-memories-of-nobody/small/blea
ch-the-movie-memories-of-nobody-s7.jpg:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:27:39 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/teru-teru-ashita/small/teru-teru-ashita-s5.jpg
:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:27:39 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/manga/mistress-fortune/small/mistress-fortune-s0.jpg
:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:27:39 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/manga/the-wild-kingdom/small/the-wild-kingdom-s3.jpg
:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:28:04 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/jigoku-shoujo/medium/jigoku-shoujo-0.jpg:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:28:05 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/jigoku-shoujo/medium/jigoku-shoujo-1.jpg:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:28:06 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/jigoku-shoujo/medium/jigoku-shoujo-00.jpg:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:28:06 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/slayers-perfect/small/slayers-perfect-s16.jpg:
0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:28:06 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/dai-mahou-touge/small/dai-mahou-touge-s0.jpg:0
:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:28:06 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/lupin-iii-il-castello-di-cagliostro/small/lupi
n-iii-il-castello-di-cagliostro-s20.jpg:0:
serf_context_run error status=20014 (Internal error)
[Tue Feb 01 21:28:06 2011] [error] [mod_pagespeed 0.9.15.3-404]
http://www.jigoku.it/images/anime/full-metal-panic-fumoffu/small/full-metal-pani
c-fumoffu-s7.jpg:0:
serf_context_run error status=20014 (Internal error)

Original comment by g.iucul...@gmail.com on 1 Feb 2011 at 8:31

GoogleCodeExporter commented 9 years ago

OK this is definitely what we should be looking at.  Note that mod_pagespeed, 
in order to rewrite your css/js/jpg/png files, must be able to fetch them.  It 
performs an HTTP fetch from the server itself.

In some installations, this is not going to work because of the way you have 
load-balancers and firewalls set up, or will simply be less efficient.  A 
solution that often helps in these situations is to make mod_pagespeed fetch 
resources from another hostname -- an IP address or 'localhost'.  In your 
configuration file you would write in your pagespeed.conf file:

   ModPagespeedMapOriginDomain localhost www.jigoku.it

Please let me know if this helps.

Original comment by jmara...@google.com on 1 Feb 2011 at 8:45

GoogleCodeExporter commented 9 years ago

Unfortunately this didn't help. After writing that I got a lot of:

[Tue Feb 01 21:49:23 2011] [error] [client ::1] File does not exist: 
/var/www/images

I suppose it searches for http://localhost/images/

Anyway I haven't any load-balancers in this server, and after stopping the 
firewall the same errors occur.

Original comment by g.iucul...@gmail.com on 1 Feb 2011 at 8:56

GoogleCodeExporter commented 9 years ago

Do you need other info?

Original comment by g.iucul...@gmail.com on 11 Feb 2011 at 5:10

GoogleCodeExporter commented 9 years ago

Other errors from log:

[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http
://www.jigoku.it/images/anime/winter-sonata/small/winter-sonata-s9.jpg
[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http
://www.jigoku.it/images/anime/ghost-hound/medium/ghost-hound-1.jpg
[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http
://www.jigoku.it/images/anime/asu-no-yoichi/small/asu-no-yoichi-s26.jpg
[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http
://www.jigoku.it/images/manga/purusatia-no-densetsu/small/purusatia-no-densetsu-
s0.jpg
[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http
://www.jigoku.it/forum/customavatars/avatar4142_1.gif
[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http
://www.jigoku.it/forum/images/smilies3/hug.gif
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot 
lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_inserts cannot 
lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_time_durat
ion_ms cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_bytes_coun
t cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_outstandin
g_count cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_timeout_co
unt cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot 
lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_inserts cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_time_duration_ms cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_bytes_count cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_outstanding_count cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http://www.jigoku.it/forum/customavatars/avatar2307_4.gif
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_inserts cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_time_duration_ms cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_bytes_count cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_outstanding_count cannot lock mutex: Invalid argument 
[Fri Feb 11 18:15:31 2011] [warn] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(780)] Fetch 
timed out: http://www.jigoku.it/forum/customavatars/avatar354_2.gif
[Fri Feb 11 18:15:31 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0211/181531:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument

Original comment by g.iucul...@gmail.com on 11 Feb 2011 at 5:19

GoogleCodeExporter commented 9 years ago

I'm also seeing this except i don't use suexec.

Error in the log : 

[Tue Feb 15 21:57:13 2011] [error] [mod_pagespeed 0.9.14.6-358] 
http://mydomain.com/wp-admin/css/login.css?ver=20100601:0: serf_context_run 
error status=20014 (Internal error)

[Tue Feb 15 21:57:13 2011] [error] [mod_pagespeed 0.9.14.6-358] 
http://www.mydomain.com/wp-admin/css/colors-fresh.css?ver=20100610:0: 
serf_context_run error status=20014 (Internal error)

Running ubuntu. latest version of modpagespeed.

Original comment by ruckafe...@gmail.com on 15 Feb 2011 at 10:20

GoogleCodeExporter commented 9 years ago

serf_context_run error status=20014 (Internal error)

I have the above error scrollnig by in my error_log and with mod_pagespeed 
installed/enabled I frequently have httpd process that get stuck chewing up 
100% CPU until I kill them off.  An strace on the stuck httpd's shows this :

(drats, lost my copy paste) but it was a fast scrolling line ending in (close)  
I'll grab it again the next time it happens.

Original comment by popow...@gmail.com on 19 Feb 2011 at 3:16

GoogleCodeExporter commented 9 years ago

If you're seeing status=20014, you're probably running off the trunk svn 
release from about a week ago.  Try upgrading to the latest trunk; we're 
curious to see if that fixes the problem.  Note that this problem has received 
a temporary fix, and we're working on a lasting fix before the next release.  
In any case, this is probably not related to the original issue reported in 
this bug report, though it is likely the cause of the problems observed by 
g.iucul..., ruckafe..., and popw...

Original comment by jmaes...@google.com on 19 Feb 2011 at 7:33

GoogleCodeExporter commented 9 years ago

I've tried version 0.9.16.3-r534, but nothing changed:

[Wed Mar 09 21:50:58 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1784] 
[0309/215058:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: http://www.jigoku.it/forum/customavatars/avatar5190_3.gif
[Wed Mar 09 21:50:58 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1784] 
[0309/215058:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: http://www.jigoku.it/forum/customavatars/avatar6093_3.gif
[Wed Mar 09 21:50:58 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1784] 
[0309/215058:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: http://www.jigoku.it/forum/customavatars/avatar5712_9.gif
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1888] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: http://www.jigoku.it/images/anime/zegapain/small/zegapain-s0.jpg
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1888] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: http://www.jigoku.it/images/anime/aki-sora/small/aki-sora-s10.jpg
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1888] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: 
http://www.jigoku.it/images/manga/angel-dust-neo/small/angel-dust-neo-s0.jpg
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1888] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: 
http://www.jigoku.it/images/manga/the-five-star-stories/small/the-five-star-stor
ies-s0.jpg
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1718] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: 
http://www.jigoku.it/images/anime/angelique-2007/small/angelique-2007-s13.jpg
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1718] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: 
http://www.jigoku.it/images/anime/duel-masters/small/duel-masters-s17.jpg
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1718] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: http://www.jigoku.it/images/manga/naruto/small/naruto-s3.jpg
[Wed Mar 09 21:50:59 2011] [warn] [mod_pagespeed 0.9.16.3-534 @1718] 
[0309/215059:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out: 
http://www.jigoku.it/images/manga/ufo-robot-goldrake/small/ufo-robot-goldrake-s1
.jpg
[Wed Mar 09 21:51:00 2011] [error] [mod_pagespeed 0.9.16.3-534 @2003] 
/var/mod_pagespeed/files/efWnsJ9_1j.lock:0: failed to stat (code=2 No such file 
or directory)
[Wed Mar 09 21:51:00 2011] [error] [mod_pagespeed 0.9.16.3-534 @2003] 
/var/mod_pagespeed/files/efWnsJ9_1j.lock:0: removing dir (code=2 No such file 
or directory)
[Wed Mar 09 21:51:04 2011] [error] [mod_pagespeed 0.9.16.3-534 @1691] 
http://www.jigoku.it/images/anime/druaga-no-tou-the-sword-of-uruk/small/druaga-n
o-tou-the-sword-of-uruk-s12.jpg:0: serf_context_run error status=110 
(Connection timed out)
[Wed Mar 09 21:51:04 2011] [error] [mod_pagespeed 0.9.16.3-534 @1691] 
http://www.jigoku.it/images/anime/kampfer/small/kampfer-s30.jpg:0: 
serf_context_run error status=110 (Connection timed out)
[Wed Mar 09 21:51:04 2011] [error] [mod_pagespeed 0.9.16.3-534 @1691] 
http://www.jigoku.it/images/anime/ikkitousen-xtreme-xecutor/small/ikkitousen-xtr
eme-xecutor-s15.jpg:0: serf_context_run error status=110 (Connection timed out)
[Wed Mar 09 21:51:04 2011] [error] [mod_pagespeed 0.9.16.3-534 @1691] 
http://www.jigoku.it/images/anime/himawari-2007/small/himawari-2007-s18.jpg:0: 
serf_context_run error status=110 (Connection timed out)
[Wed Mar 09 21:51:04 2011] [error] [mod_pagespeed 0.9.16.3-534 @1691] 
http://www.jigoku.it/images/manga/detective-conan/small/detective-conan-s2.jpg:0
: serf_context_run error status=110 (Connection timed out)
[Wed Mar 09 21:51:04 2011] [error] [mod_pagespeed 0.9.16.3-534 @1691] 
http://www.jigoku.it/images/manga/sempre-piu-blu/small/sempre-piu-blu-s0.jpg:0: 
serf_context_run error status=110 (Connection timed out)

Original comment by g.iucul...@gmail.com on 9 Mar 2011 at 8:53

GoogleCodeExporter commented 9 years ago

[deleted comment]

GoogleCodeExporter commented 9 years ago

Hi, 

I am getting same "fetch time out" and "status-20014" errors above and 
experiencing very high CPU load with many httpd processes...after a certain 
period of time the server becomes unresponsive and I need a reboot.
Centos 5.5
Apache/2.2.3 prefork MPM
mod_pagespeed 0.9.15.3-404

Original comment by 3dl...@gmail.com on 10 Mar 2011 at 4:29

GoogleCodeExporter commented 9 years ago

same here, mod_php
then i got a trace on the log

*** glibc detected *** /usr/sbin/apache2: double free or corruption (out): 
0x00007f235a95f670 ***
PHP Fatal error:  Call to undefined function count() in 
/var/www//ips_kernel/classDb.php on line 1916
PHP Fatal error:  Call to undefined function count() in 
/var/www//ips_kernel/classDb.php on line 1916
*** glibc detected *** /usr/sbin/apache2: double free or corruption (out): 
0x00007f2359b87360 ***
PHP Fatal error:  Call to undefined function count() in 
/var/www//ips_kernel/classDb.php on line 1916

Original comment by diazl...@gmail.com on 23 Apr 2011 at 6:02

GoogleCodeExporter commented 9 years ago

I'm not sure if all the different reports in this issue are the same or not.

diazl:  what's making you think this is related to mod_pagespeed?  Could you 
investigate 'Call to undefined function count()'?

3dl and others:  If you could upgrade to the latest binary, 0.9.16.9, that 
might help.  There are a variety of interoperability issues with other Apache 
modules that were resolved in that release, including mod_speling, 
mod_includes, and more nuances of mod_rewrite.

The the first thing to resolve is all the Serf fetch failures in the logs.  
Resolving these generally involves usage of ModPagespeedMapOriginDomain.  But 
this is one of the harder things for me to debug remotely because it's really a 
function of the network view frmo the machine running Apache.

Original comment by jmara...@google.com on 24 Apr 2011 at 8:52

GoogleCodeExporter commented 9 years ago

Upgraded to 0.9.16.9 now I've a lot of segfault:

[Thu Apr 28 14:44:09 2011] [notice] child pid 8039 exit signal Segmentation 
fault (11)
[Thu Apr 28 14:44:09 2011] [notice] child pid 8041 exit signal Segmentation 
fault (11)
[Thu Apr 28 14:44:09 2011] [notice] child pid 8042 exit signal Segmentation 
fault (11)
[Thu Apr 28 14:44:09 2011] [notice] child pid 8045 exit signal Segmentation 
fault (11)
[Thu Apr 28 14:44:09 2011] [notice] child pid 8052 exit signal Segmentation 
fault (11)
[Thu Apr 28 14:44:09 2011] [notice] child pid 8054 exit signal Segmentation 
fault (11)
[Thu Apr 28 14:44:09 2011] [notice] Graceful restart requested, doing restart
[Thu Apr 28 14:44:14 2011] [warn] [mod_pagespeed 0.9.16.9-576 @6066] 
[0428/144414:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out
: 
http://www.jigoku.it/images/anime/operation-sanctuary/small/operation-sanctuary-
s16.jpg
[Thu Apr 28 14:44:16 2011] [warn] [mod_pagespeed 0.9.16.9-576 @6066] 
[0428/144416:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out
: http://www.jigoku.it/images/anime/sanpei/small/sanpei-s8.jpg
[Thu Apr 28 14:44:16 2011] [warn] [mod_pagespeed 0.9.16.9-576 @6066] 
[0428/144416:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out
: 
http://www.jigoku.it/images/anime/macross-frontier/small/macross-frontier-s14.jp
g
[Thu Apr 28 14:44:16 2011] [warn] [mod_pagespeed 0.9.16.9-576 @6066] 
[0428/144416:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(867)] Fetch 
timed out
: http://www.jigoku.it/images/anime/azumanga-daioh/small/azumanga-daioh-s28.jpg

Original comment by g.iucul...@gmail.com on 28 Apr 2011 at 12:52

GoogleCodeExporter commented 9 years ago

The seg-faults are something we should definitely look at.  I wish there was a 
way we could reproduce.  Is there any way that you could take a new machine (or 
VM) install Apache, PHP, mod_pagespeed, and a tiny piece of your site, repro 
your problem, and then send us the entire server setup?  I realize this is not 
a trivial request but it might be the best way we have to get to the bottom of 
this.  I really appreciate it!

I'm still mystified why the serf fetches are timing out, though.  I feel it 
must be related to the redirection being done within PHP.  Unfortunately I know 
almost nothing about PHP or why it would do redirection.  But if we could repro 
your setup?

The fetches that we are doing from within the server process are supposed to 
behave identically to those that would be initiated by your users' browsers to 
fetch your images, etc.  Is there something that PHP might be doing to get in 
the way of that?

Original comment by jmara...@google.com on 28 Apr 2011 at 1:08

GoogleCodeExporter commented 9 years ago

My setup is php+fastcgi+suexec+mod-vhost-hash-alias, you can find a simple 
vhost conf in the first post.
If you need I could prepare a VM to share my setup.

Original comment by g.iucul...@gmail.com on 28 Apr 2011 at 1:19

GoogleCodeExporter commented 9 years ago

quemu/kvm image is ready, you can download it: 
http://ctu.iuculano.it/modpagespeed/modpagespeed_bug.img.tar.gz

this is a Debian installation with my web setup. 

In /home/web/httpdocs/www.testdomain.bogus there is a simple joomla 
installation that causes segfault with modpagespeed 
(/var/log/apache2/error.log).

root password: testbug

Let me know if you need further information.

Original comment by g.iucul...@gmail.com on 28 Apr 2011 at 4:27

GoogleCodeExporter commented 9 years ago

That's great!  Thanks!  We'll take a look.

Original comment by jmara...@google.com on 28 Apr 2011 at 4:37

GoogleCodeExporter commented 9 years ago

Just to update you, I couldn't get the image to boot Linux (that is, grub 
starts, but not the OS), but I'll see if I can figure things out based on the 
configuration.

Original comment by morlov...@google.com on 9 May 2011 at 2:44

GoogleCodeExporter commented 9 years ago

hmm, this is a 64bit OS image, I don't remember if a 32bit quemu/kvm host can 
run it. Have you tried with a 64bit or 32 bit quemu/kvm host?

Original comment by g.iucul...@gmail.com on 9 May 2011 at 2:53

GoogleCodeExporter commented 9 years ago

Any progress on this? I tried 0.9.17.7-r716 but I've a lot of Segmentation 
fault:

[Tue Jul 12 14:34:04 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9347] 
[0712/143404:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/forum/images/smilies2/biggrin.gif
[Tue Jul 12 14:34:04 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9426] 
[0712/143404:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/layout/news/commenta.jpg
[Tue Jul 12 14:34:04 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9426] 
[0712/143404:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/forum/customavatars/avatar5760_1.gif
[Tue Jul 12 14:34:04 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9426] 
[0712/143404:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/forum/customavatars/avatar5925_1.gif
[Tue Jul 12 14:34:05 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9347] 
[0712/143405:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: 
http://www.jigoku.it/forum/images/v2/statusicon/user_online.gif
[Tue Jul 12 14:34:05 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9426] 
[0712/143405:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/forum/customavatars/avatar4882_17.gif
[Tue Jul 12 14:34:05 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9347] 
[0712/143405:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/forum/customavatars/avatar6766_1.gif
[Tue Jul 12 14:34:05 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9377] 
[0712/143405:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: 
http://www.jigoku.it/images/anime/mai-otome-zwei/small/mai-otome-zwei-s20.jpg
[Tue Jul 12 14:34:05 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9377] 
[0712/143405:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/images/anime/tokko/small/tokko-s15.jpg
[Tue Jul 12 14:34:05 2011] [notice] child pid 9347 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:05 2011] [notice] child pid 9395 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:05 2011] [notice] child pid 9539 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:05 2011] [notice] child pid 9543 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:05 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9426] 
[0712/143405:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: http://www.jigoku.it/forum/images/smilies3/asd.gif
[Tue Jul 12 14:34:05 2011] [warn] [mod_pagespeed 0.9.17.7-716 @9377] 
[0712/143405:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(905
)] Fetch timed out: 
http://www.jigoku.it/images/manga/layla-e-rei/small/layla-e-rei-s1.jpg
[Tue Jul 12 14:34:06 2011] [notice] child pid 9377 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:06 2011] [notice] child pid 9465 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:06 2011] [notice] child pid 9565 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:06 2011] [notice] child pid 9567 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:06 2011] [notice] child pid 9568 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:06 2011] [notice] child pid 9633 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:07 2011] [notice] child pid 9426 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:07 2011] [notice] child pid 9570 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:07 2011] [notice] child pid 9631 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:07 2011] [notice] child pid 9635 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:07 2011] [notice] child pid 9639 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:07 2011] [notice] child pid 9641 exit signal Segmentation 
fault (11)
[Tue Jul 12 14:34:07 2011] [notice] child pid 9646 exit signal Segmentation 
fault (11)

Original comment by g.iucul...@gmail.com on 12 Jul 2011 at 12:37

GoogleCodeExporter commented 9 years ago

Hi, 

Update: I am still getting the error: 

mod_pagespeed 0.9.18.7-900 @32326] 
[1012/051555:WARNING:net/instaweb/apache/serf_url_async_fetcher.cc(915)] Fetch 
timed out http://www.example.com/image.jpg
[notice] child pid 32025 exit signal Segmentation fault (11)

...even though I've disabled the rewrite_images with this on pagespeed.conf

ModPagespeedDisableFilters rewrite_images

now I'm on

CentOS Linux release 6.0  (64-bit)
Apache/2.2.15
mod-pagespeed 0.9.18.7-900

Original comment by 3dl...@gmail.com on 12 Oct 2011 at 5:57

GoogleCodeExporter commented 9 years ago

Hi -- are you still getting this issue with the latest release of mod_pagespeed 
(0.10.21.2)?  We've had a fair amount of improvements to the code that was 
giving you trouble since 0.9.18.7.

Thanks.

Original comment by jmara...@google.com on 24 May 2012 at 7:20

Changed state: RequestClarification

GoogleCodeExporter commented 9 years ago

I am getting this issue with Mod-Pagespeed: (0.10.21.2-1381)
I get a lot of 
[warn] [mod_pagespeed 0.10.21.2-1381 @7867] 
[0714/225255:WARNING:serf_url_async_fetcher.cc(1050)] Fetch timed out: 
http://holidayiq.com/images/stumbleupon.gif (2 waiting for 50ms)

and then the server load increases rapidly.
I was trying to move pagespeed file based cache to memory
 Any pointers to this issue?

Original comment by mo...@holidayiq.biz on 14 Jul 2012 at 5:59

GoogleCodeExporter commented 9 years ago

Maybe try setting ModPagespeedFetcherTimeoutMs:

https://developers.google.com/speed/docs/mod_pagespeed/system#Setting%20the%20UR
L%20fetcher%20timeout

Although there is probably an underlying issue that is causing fetches to 
timeout.

Original comment by sligocki@google.com on 16 Jul 2012 at 4:44

GoogleCodeExporter commented 9 years ago

Not sure the status of this, but 
[Tue Feb 01 21:49:23 2011] [error] [client ::1] File does not exist: 
/var/www/images

looks like it is trying to connect to an IPv6 address

Original comment by yolabi...@gmail.com on 28 Nov 2013 at 1:37