Apache stuck indefinitely waiting for PSOL

[GoogleCodeExporter]

I just installed mod_pagespeed on my centos 7 and got tons of httpd errors log 
in 1 minute. An example line of error:
<code>
[Tue Feb 10 11:05:14.311755 2015] [pagespeed:warn] [pid 21132:tid 
139634310850304] [mod_pagespeed 1.9.32.3-4448 @21132] Waiting for completion of 
URL http://exampledomain.com/example-slug/ for 45.001 sec
</code>

ALL requests got error, include image requests too.

My server hardware specs:
* Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz, 8 cores
* 32 GB DDR3 RAM
* 2 x 2 TB SATA 6 Gb/s 7200 rpm HDD (Software-RAID 1) Class Enterprise

Software specs:
Operating system: CentOS Linux 7.0.1406
Kernel: Linux 3.10.0-123.20.1.el7.x86_64 on x86_64

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 
PHP/5.6.5 mod_perl/2.0.9dev Perl/v5.16.3
Server MPM: event

What version of the product are you using (please check X-Mod-Pagespeed
header)?
mod-pagespeed-stable-1.9.32.3-4448.x86_64

URL of broken page:
I removed module after 1 minute of terror. If a google developer want to learn 
more of server information, mail me an ip address that i can give permission to 
look mod_info print.

Original issue reported on code.google.com by unsalkor...@gmail.com on 10 Feb 2015 at 9:35

[jeffkaufman]

We had a meeting offline to work on this bug. Some things that came out of that:

• We're not stuck on a slow blocking call because that would have shown up in the backtraces we got from gdb.
• It's probably some rare situation in which a callback is being missed, most likely because of a race.
• What that same situation happens in Nginx the equivalent of ApacheFetch doesn't exist and so Nginx times out the request instead of tying up an Apache process indefinitely.
• One fix would be to make ApacheFetch::Wait() time out and log an error after a long enough wait.
• This would not fix the underlying bug, but would put Apache into the same situation as Nginx, where the bug is rare and affects only that one request.

Ideas for more places to look for the underlying bug:

• IPRO has a CacheUrlAsyncFetcher, which might be using vestigial fetch locks; what happens if we fail to get the lock because two requests come in right together?
• We have locking to avoid optimizing a resource multiple times at once. What happens if we fail to get that lock in the ipro flow?
• AsyncFetchWithLock::Start is suspicious, but I don't remember seeing "is already being fetched" in people's logs

[eldk]

Hello,

I have made some tests this night with IPRO ON.

As I "understand", IPRO may rewrite images when direct access it/not asked in HTML. For the first call, if image is not in the client browser cache, a copy of original image is done in the folder /var/cache/mod_pagespeed/domain/imagefolder/.

After n next calls with 200 apache answer, a rewrited minimized ... image is created and then served to the client, If a 304 answer is given, nothing is done (for example google images crawl or image in browser cache).

So I still have the "waiting for completion of URL" warning. But rewrited images are created in cache, served to the client with good type, webp (Chrome) or jpg (Firefox). There is a warning, but rewriting have been well done.

For those test, I have stopped apache, renamed /var/cache/mod_pagespeed/ folder, then created a new one which is readable and writable by apache user and group. At restart, apache recreated the domain subfolder, may read and write to it. But I can't find any rname folder. Is this folder absolutely needed and what is creating it ? Where should I find a doc on what is rname folder ?

Greatings,

Eric

Version: 13: on

Filters cw Collapse Whitespace gp Convert Gif to Png jp Convert Jpeg to Progressive jw Convert Jpeg To Webp pj Convert Png to Jpeg hw Flushes html ci Inline Css io In-place optimize for browser js Jpeg Subsampling rj Recompress Jpeg rp Recompress Png rw Recompress Webp rc Remove Comments cf Rewrite Css jm Rewrite External Javascript jj Rewrite Inline Javascript cp Strip Image Color Profiles md Strip Image Meta Data

Options EnableCachePurge (euci) False EnableRewriting (e) 1 FileCacheCleanIntervalMs (afcci) 3600000 FileCacheInodeLimit (afcl) 500000 FileCachePath (afcp) /var/cache/mod_pagespeed/ FileCacheSizeKb (afc) 2048000 InPlaceResourceOptimization (ipro) True LogDir (ald) /var/log/pagespeed RewriteLevel (l) Optimize For Bandwidth SslCertDirectory (assld) /etc/ssl/certs StatisticsLogging (asle) True

Domain Lawyer

Invalidation Timestamp: (none)

[jeffkaufman]

So I still have the "waiting for completion of URL" warning. But rewrited images are created in cache, served to the client with good type, webp (Chrome) or jpg (Firefox). There is a warning, but rewriting have been well done.

When you were testing this were there any responses that hung or otherwise didn't return a valid image?

When you got the "waiting for completion of URL" message, how many times did it show up? Just a few or was it printing every 5s until you shut down the server?

For those test, I have stopped apache, renamed /var/cache/mod_pagespeed/ folder, then created a new one which is readable and writable by apache user and group. At restart, apache recreated the domain subfolder, may read and write to it. But I can't find any rname folder. Is this folder absolutely needed and what is creating it ? Where should I find a doc on what is rname folder ?

Apache will recreate the rname folder when it's needed. That's the metadata cache, storing information about things like "when you optimize the image with url X you get the image we've stored at url Y in the http cache".

In general, the best way to flush the pagespeed cache is to tell pagespeed to do the flushing: https://developers.google.com/speed/pagespeed/module/system#flush_cache

[eldk]

Hello,

When you were testing this were there any responses that hung or otherwise didn't return a valid image?

No, no hung, always valid images.

When you got the "waiting for completion of URL" message, how many times did it show up? Just a few or was it printing every 5s until you shut down the server?

I'm using FCGID so I think no task from apache could hang indefinitely without been forced to close. The "waiting for completion " could occure the first time the image is called, and for what I have seen being repeted a few times until 450 secondes max.

When an image is in cache, the message should come back later thus the cached image is served. One response header for that case in Firefox:` ... Date: Thu, 07 May 2015 16:34:36 GMT Content-Length: 1256601 Accept-Ranges: bytes Etag: W/"PSA-aj-oe4DUPjTLN" Expires: Tue, 03 Nov 2015 03:54:37 GMT Cache-Control: max-age=15506400 Vary: Accept Last-Modified: Thu, 07 May 2015 16:34:36 GMT X-Content-Type-Options: nosniff Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: image/jpeg 200 OK ... Content- lenght is the same as rewrited image and the etag is the one stored in the rewrite image file name : 11256876 mai 7 05:55 xyeux-anguille-anguilla-anguilla.jpg.pagespeed.ic.oe4DUPjTLN.jpg,

[Thu May 07 18:35:55 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @13509] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/jpg/yeux-anguille-anguilla-anguilla.jpg for 10.025 sec

at 18h41 GMT+2 no file has been recreated or modified.

Greatings,

Eric

[eldk]

Hello again,

I found now some message "waiting ..." for some images that have been "Serfed" due to embedded in HTML.

Here it's the same, good resized image (Etag) served but warning message.

grep in access log :`

[07/May/2015:18:18:56 +0200] "GET /IMG/cache-300x200/IMG_3711_v1-300x200.jpg HTTP/1.1" 200 18060 "http://www.google.fr/" "Mozilla/5.0 (Linux; Android 4.4.2; SM-G800F Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.109 Mobile Safari/537.36"

It's a direct access to image, from a google images search request :

That create this cached image :

7580 mai 7 18:18 xIMG_3711_v1-300x200.jpg.pagespeed.ic.1JC-g9sPYj.webp,

So this image wasn't in cache and hasn't been "Serfed" in the webp format until this request was made.

Greatings, Eric

[jeffkaufman]

Since this is probably caused by the IPRO flow in PSOL not calling a callback under unusual circumstances, I went through and traced the callbacks in this flow. All IPRO requests go through three callbacks, where the last callback differs between 1st/2nd requests and 3+rd requests:

• RewriteContext::Fetch calls RewriteDriver::AddRewriteTask to get RewriteContext::StartFetch called on the rewrite thread (or RewriteContext::CancelFetch if cancelled).
• RewriteContext::StartFetchImpl calls Get() on the metadata cache, to get RewriteContext::FetchCacheDone called with the results of the metadata cache lookup.
• First (nothing in cache) or second request (recorded resource in cache)
  • CacheUrlAsyncFetcher::Fetch calls Find() on the http cache to get CacheFindCallback::Done called with the results of the http cache lookup.
• Third (optimized resource in cache):
  • RewriteContext::FetchTryFallback calls Find() on the http cache to get RewriteContext::FetchFallbackCacheDone called with the results of the http cache lookup

[eldk]

Hello,

Thus images are all rewrited (jpg and webp). There are still "waiting for completion .." messages. For example : Call to this image (always direct access to image) : Apache access log : [08/May/2015:11:03:19 +0200] "GET /IMG/montage_1_clipot_creux.jpg HTTP/1.1" 200 683251 "-" "Mozilla/5.0 (Linux; Android 4.4.2; K00Z Build/KVT49L) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Safari/537.36 GSA/4.4.11.16.x86" For which all have been rewrited : 1529415 mai 7 08:54 montage_1_clipot_creux.jpg, 476402 mai 7 09:20 xmontage_1_clipot_creux.jpg.pagespeed.ic.57rrAy504D.jpg, 481435 mai 7 18:18 xmontage_1_clipot_creux.jpg.pagespeed.ic.bljlr_GZ6w.webp, 683124 mai 7 09:35 xmontage_1_clipot_creux.jpg.pagespeed.ic.jAD6QblrdB.jpg, 350413 mai 7 12:30 xmontage_1_clipot_creux.jpg.pagespeed.ic.XsQS4awGxO.webp, Still spawn warnings : [Fri May 08 11:03:24 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 5 sec [Fri May 08 11:03:29 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 10.005 sec [Fri May 08 11:03:34 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 15.015 sec [Fri May 08 11:03:39 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 20.015 sec [Fri May 08 11:03:44 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 25.063 sec [Fri May 08 11:03:49 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 30.16 sec [Fri May 08 11:03:54 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 35.16 sec [Fri May 08 11:04:00 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 40.28 sec [Fri May 08 11:04:05 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 45.281 sec [Fri May 08 11:04:10 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 50.281 sec [Fri May 08 11:04:15 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @32387] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/montage_1_clipot_creux.jpg for 55.297 sec

Due to this and precedent post, First (nothing in cache/recorded resource in cache) and Third are concerned.

I will follow the logs for a most popular image to see if it's happened for every call when in Third case then remove the cache for that image and do the same for First case (nothing in cache/recorded resource in cache).

Greatings,

Eric

[Dead2]

I hit this one on a busy server tonight. We do not use memcached or InPlaceResourceOptimization. We do use LoadFromFileMatch and ModPagespeedMapOriginDomain.

One of the vhosts that experienced this problem tonight has the following config Inherited:

    ModPagespeed on
    ModPagespeedRespectXForwardedProto on
    ModPagespeedRewriteLevel PassThrough
    ModPagespeedForbidAllDisabledFilters false
    ModPagespeedDefaultSharedMemoryCacheKB 100000

    ModPagespeedLoadFromFileRuleMatch disallow .*
    ModPagespeedLoadFromFileRuleMatch allow \.(css|js|png|jpg|webp|gif|ico|ttf|woff)$

    ModPagespeedNumRewriteThreads 3
    ModPagespeedNumExpensiveRewriteThreads 1
    ModPagespeedRewriteRandomDropPercentage 10
    ModPagespeedRewriteDeadlinePerFlushMs 15

vhost-spesific:

  ModPagespeed on
  ModPagespeedFileCachePath '/kunder/xx/zz/pagespeed-cache'
  ModPagespeedMapOriginDomain 'http://domain.com:8000' 'http://domain.com'
  ModPagespeedMapOriginDomain 'http://domain.com:8000' 'https://domain.com'
  ModPagespeedLoadFromFile 'http://domain.com/' '/kunder/xx/yy/public'
  ModPagespeedCombineAcrossPaths on
  ModPagespeedEnableFilters recompress_jpeg,recompress_png,convert_to_webp_lossless,collapse_whitespace,elide_attributes,trim_urls,prioritize_critical_css,remove_quotes,remove_comments,rewrite_javascript,inline_javascript,combine_javascript,combine_css,rewrite_style_attributes,move_css_to_head,move_css_above_scripts,fallback_rewrite_css_urls,rewrite_css,inline_import_to_link,inline_css,flatten_css_imports,sprite_images,insert_image_dimensions,resize_images,recompress_webp,convert_jpeg_to_webp,convert_jpeg_to_progressive,inline_preview_images,resize_mobile_images,dedup_inlined_images,insert_dns_prefetch

The above config lines have been modified obviously. There are several more MapOriginDomain and LoadFromFile (all pointing to same folder and to the same source url).

I might be able to email the configfiles and logfiles on request, it has a loglevel of 'notify' so it is quite verbose (although nothing really seems to stand out that I could see).

I wanted to check how many unique processes had been stuck in the waiting-game, turns out it was 22 processes.

# grep "Waiting" error_log|cut -d "]" -f 3 |sort|uniq -c
    504  [pid 100178
    493  [pid 101997
    476  [pid 104958
    475  [pid 105131
    410  [pid 115976
    398  [pid 118044
    355  [pid 125169
    354  [pid 125372
    327  [pid 129888
    327  [pid 129962
    314  [pid 132203
    307  [pid 133363
    295  [pid 135338
    727  [pid 92559
    650  [pid 93317
    575  [pid 93910
    557  [pid 94091
    555  [pid 94143
    551  [pid 94313
    534  [pid 94926
    528  [pid 95840
    511  [pid 98948

First occurence at 05:11, next process at 05:23, next 05:25, next 05:27. First three are different vhosts, the last one is same as the second one. Have been running for days without such problems (have not checked further back). At 05:59 the monitoring system woke me up due to multiple vhosts not working properly and I restarted apache, no errors have appeared since.

So, it looks to me that once one has been triggered, more will soon follow.

[jmarantz]

Hans -- you mean you saw the "Waiting for Completion" spinning message in your log file? And you don't have ipro on, and you don't have MapProxyDomain either?

Can you paste the exact message you saw?

-Josh

On Fri, May 8, 2015 at 7:16 AM, Hans Kristian Rosbach < notifications@github.com> wrote:

I hit this one on a busy server tonight. We do not use memcached or InPlaceResourceOptimization. We do use LoadFromFileMatch and ModPagespeedMapOriginDomain.

One of the vhosts that experienced this problem tonight has the following config Inherited: ModPagespeed on ModPagespeedRespectXForwardedProto on ModPagespeedRewriteLevel PassThrough ModPagespeedForbidAllDisabledFilters false ModPagespeedDefaultSharedMemoryCacheKB 100000

ModPagespeedLoadFromFileRuleMatch disallow .* ModPagespeedLoadFromFileRuleMatch allow .(css|js|png|jpg|webp|gif|ico|ttf|woff)$

ModPagespeedNumRewriteThreads 3 ModPagespeedNumExpensiveRewriteThreads 1 ModPagespeedRewriteRandomDropPercentage 10 ModPagespeedRewriteDeadlinePerFlushMs 15

vhost-spesific: ModPagespeed on ModPagespeedFileCachePath '/kunder/xx/zz/pagespeed-cache' ModPagespeedMapOriginDomain 'http://domain.com:8000' 'http://domain.com' ModPagespeedMapOriginDomain 'http://domain.com:8000' 'https://domain.com' ModPagespeedLoadFromFile 'http://domain.com/' '/kunder/xx/yy/public' ModPagespeedCombineAcrossPaths on ModPagespeedEnableFilters recompress_jpeg,recompress_png,convert_to_webp_lossless,collapse_whitespace,elide_attributes,trim_urls,prioritize_critical_css,remove_quotes,remove_comments,rewrite_javascript,inline_javascript,combine_javascript,combine_css,rewrite_style_attributes,move_css_to_head,move_css_above_scripts,fallback_rewrite_css_urls,rewrite_css,inline_import_to_link,inline_css,flatten_css_imports,sprite_images,insert_image_dimensions,resize_images,recompress_webp,convert_jpeg_to_webp,convert_jpeg_to_progressive,inline_preview_images,resize_mobile_images,dedup_inlined_images,insert_dns_prefetch

The above config lines have been modified obviously. There are several more MapOriginDomain and LoadFromFile (all pointing to same folder and to the same source url).

I might be able to email the configfiles and logfiles on request, it has a loglevel of 'notify' so it is quite verbose (although nothing really seems to stand out that I could see).

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/mod_pagespeed/issues/1048#issuecomment-100197419 .

[Dead2]

That is right, no MapProxyDomain and no ipro. I even did a grep on all config files now just in case, they are not and cannot (by a non-root user) be enabled on this server.

The initial four warnings:

[Fri May 08 05:11:09.940940 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 5 sec
[Fri May 08 05:11:14.941139 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 10.001 sec
[Fri May 08 05:11:19.941315 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 15.001 sec
[Fri May 08 05:11:24.941497 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 20.001 sec

[jmarantz]

Oh, sorry for the confusion. You are using 1.9, so unless you turn it off specifically, you have ipro. TO turn it off, use

ModPagespeedInPlaceResourceOptimization off

On Fri, May 8, 2015 at 8:33 AM, Hans Kristian Rosbach < notifications@github.com> wrote:

That is right, no MapProxyDomain and no ipro. I even did a grep on all config files now just in case, they are not and cannot (by a non-root user) be enabled on this server.

The initial four warnings:

[Fri May 08 05:11:09.940940 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 5 sec [Fri May 08 05:11:14.941139 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 10.001 sec [Fri May 08 05:11:19.941315 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 15.001 sec [Fri May 08 05:11:24.941497 2015] [pagespeed:warn] [pid 92559] [mod_pagespeed 1.9.32.3-4448 @92559] Waiting for completion of URL http://domain.com/wp-content/uploads/2014/06/11.jpg for 20.001 sec

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/mod_pagespeed/issues/1048#issuecomment-100218658 .

[Dead2]

I have seen .jpg, .cs and .js files in waiting.

One strange thing though is that the url it is waiting to download from is not the one specified in the config files using ModPagespeedMapOriginDomain.

So at least according to the message, it is attempting to download the file from the frontend nginx instead of the backend apache at port 8000. This would then loop since nginx asks apache, and possibly nginx is waiting for the previous (the initial) request for that file to complete so it can deliver a cached version..?

Or is it just a wrong error message, just displaying the wrong url?

[Dead2]

@jmarantz Ah, now that you mention it, I remember seeing that in the changelog. So yes, then ipro is probably enabled.

[jmarantz]

The URL is printed in the error message prior to mapping it with MapOriginDomain, so I'm not worried about that.

It looks like based on your directives, http://domain.com/wp-content/uploads/2014/06/11.jpg should be stored in '/kunder/xx/yy/public/wp-content/uploads/2014/06/11.jpg'. Is it there?

I wonder if this is an interaction between ipro and URLs which match our LoadFromFile path, but are missing.

In the non-ipro case, we would simply not optimize such resources.

Note that we do not fallback to http fetches when a URL in the load-from-file path is missing; we just fail to do the fetch.

-Josh

On Fri, May 8, 2015 at 8:40 AM, Hans Kristian Rosbach < notifications@github.com> wrote:

I have seen .jpg, .cs and .js files in waiting.

One strange thing though is that the url it is waiting to download from is not the one specified in the config files using ModPagespeedMapOriginDomain.

So at least according to the message, it is attempting to download the file from the frontend nginx instead of the backend apache at port 8000. This would then loop since nginx asks apache, and possibly nginx is waiting for the previous (the initial) request for that file to complete so it can deliver a cached version..?

Or is it just a wrong error message, just displaying the wrong url?

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/mod_pagespeed/issues/1048#issuecomment-100221403 .

[Dead2]

The file exists and has been there for 7 months.

[Dead2]

It appears that not only static files are affected. These are the variations that failed, including 3 php scripts and those dont match the LoadFromFileRuleMatch.

http://domain1.com/wp-content/uploads/2014/06/11.jpg http://domain2.com/?page=30 http://domain3.com/wp-content/plugins/gravityforms/js/jquery.maskedinput-1.3.1.min.js?ver=1.8.17 http://domain4.com/wp-includes/css/admin-bar.min.css?ver=4.1.2 http://domain5.com/wp-content/themes/feast/images/miljofyrtaarn.png http://domain5.com/wp-content/themes/feast/images/prevver.png http://domain6.com/nettbutikk-2/?filter_typekaffe=82&filter_land=104&filter_storrelse=84,85,86 http://domain7.com/

[eldk]

Hello,

I'm going on the tests.

Should a restart of apache make the cache rewriting images again, thus the rewrited images were in the cache ?

Greatings.

Eric

[jeffkaufman]

@eldk I'm not sure if this is what you're asking, but no, restarting Apache won't clear the cache. To do that you need https://developers.google.com/speed/pagespeed/module/system#flush_cache

[eldk]

Hello, @jeffkaufman I try to be more understandable.

I want to know, if when apache is restarted, the images in the cache, the original one, and the rewrited images (2 jpg + 2 webp) are rewrited again (not flush but rewrited).

a ls -al nameofimage give the time of modification of the image file. When I restart apache (sudo service apache2 restart), the time of modification is update as if the images were rewrited again and I think they are.

Is it the expected behaviour ?

Greatings,

Eric

[eldk]

Hello, I have changes some parameters in my mod_pagespeed.conf, use of ModPagespeedLoadFromFile for my image directory.

I have less "waiting from completion ..." messages. The message that give more than 5 seconds waiting time are less than 90%.

Those messages which give longer waiting times seem to have common things :

• thus the apache log give a 200 answer for the request, the size of the file is not good : less than the size of original and rewrited images.
• when the user, push a new request on the same image (I think content is not well displayed), pagespeed throw a new process that should last 5 seconds or less, and sometimes never end (killed by FCGID). And so on on each try ...
• The more errors appears with this user agent : "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36" - On ten check in apache access log, 9/10 errors.
• it should happened on getting the cached image or when creating the cached image
• when creating the cached image, it could last ... without creating cached image

I can't reproduce it myself for the moment (will make a try with google pagespeed insight).

Greatings,

Eric

Version: 13: on Filters cw Collapse Whitespace gp Convert Gif to Png jp Convert Jpeg to Progressive jw Convert Jpeg To Webp pj Convert Png to Jpeg hw Flushes html ci Inline Css io In-place optimize for browser js Jpeg Subsampling rj Recompress Jpeg rp Recompress Png rw Recompress Webp rc Remove Comments cf Rewrite Css jm Rewrite External Javascript jj Rewrite Inline Javascript cp Strip Image Color Profiles md Strip Image Meta Data Options EnableCachePurge (euci) False EnableRewriting (e) 1 FileCacheCleanIntervalMs (afcci) 3600000 FileCacheInodeLimit (afcl) 500000 FileCachePath (afcp) /var/cache/mod_pagespeed/ FileCacheSizeKb (afc) 4096000 ImplicitCacheTtlMs (ict) 15549367000 InPlaceResourceOptimization (ipro) True LoadFromFileCacheTtlMs (lfct) 15549367000 LogDir (ald) /var/log/pagespeed RewriteLevel (l) Optimize For Bandwidth SslCertDirectory (assld) /etc/ssl/certs StatisticsLogging (asle) True

[jeffkaufman]

the size of the file is not good : less than the size of original and rewrited images.

Are you saying that according to Apache logs, when a request gets into this "waiting for completion of url" state, that it (a) does serve out a response and (b) the bytes in the response are >0 but less than original or optimized? That's very surprising, but if that's what's happening then it's also very good debugging information.

EDIT: And, this is for IPRO resources? That is, js/css/images that PageSpeed rewrites in-place, without moving?

[eldk]

Hello,

Yes, it's with IPRO. limited to some image folder (no css/js ..) only jpg (more than 90%), gif, and png. There is some PDF but they are not rewritten.

Warning messages always occure to jpg (thus there is a few gif with very high hits).

Yes, (a) a response is given (status 200 in apache access log), (b) the size is > 0 but < to size of original and rewrited images (far from ).

Greatings,

Eric

[eldk]

One example :` File sizes : Original : 1944696 déc. 20 2012 coucher_soleil_surf_casting_plage_de_vert_bois.jpg Resized : 252359 mai 12 05:23 xcoucher_soleil_surf_casting_plage_de_vert_bois.jpg.pagespeed.ic.3yu50AYt9t.webp, 689914 mai 12 03:44 xcoucher_soleil_surf_casting_plage_de_vert_bois.jpg.pagespeed.ic.o9Im1AbLI-.jpg, 382387 mai 12 03:43 xcoucher_soleil_surf_casting_plage_de_vert_bois.jpg.pagespeed.ic.oSN3mSp6D3.webp, 431269 mai 12 07:50 xcoucher_soleil_surf_casting_plage_de_vert_bois.jpg.pagespeed.ic.XKOxTrFFXo.jpg,

Apache access log (the same user - sameip - that I think try to reload image): sameip - - [12/May/2015:11:14:35 +0200] "GET /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg HTTP/1.1" 200 94384 "-" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G316HU Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 GSA/3.6.14.1337016.arm" sameip - - [12/May/2015:11:20:43 +0200] "GET /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg HTTP/1.1" 200 80504 "-" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G316HU Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 GSA/3.6.14.1337016.arm" sameip - - [12/May/2015:11:22:36 +0200] "GET /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg HTTP/1.1" 200 101324 "-" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G316HU Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 GSA/3.6.14.1337016.arm" sameip - - [12/May/2015:11:28:36 +0200] "GET /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg HTTP/1.1" 200 92996 "-" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G316HU Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 GSA/3.6.14.1337016.arm" sameip - - [12/May/2015:11:30:25 +0200] "GET /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg HTTP/1.1" 200 95772 "-" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G316HU Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 GSA/3.6.14.1337016.arm" sameip - - [12/May/2015:11:32:22 +0200] "GET /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg HTTP/1.1" 200 102712 "-" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G316HU Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 GSA/3.6.14.1337016.arm"

And some of the warnings (there are multi task number because user try to reload the image, and new process are thrown) :

[Tue May 12 11:30:02 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 75.247 sec [Tue May 12 11:30:02 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14068] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 435.955 sec [Tue May 12 11:30:05 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @13871] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 551.309 sec [Tue May 12 11:30:07 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 80.267 sec [Tue May 12 11:30:07 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14068] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 440.955 sec [Tue May 12 11:30:10 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @13871] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 556.309 sec [Tue May 12 11:30:12 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 85.274 sec [Tue May 12 11:30:12 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14068] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 445.956 sec [Tue May 12 11:30:17 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 90.275 sec [Tue May 12 11:30:17 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14068] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 450.956 sec [Tue May 12 11:30:22 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 95.275 sec [Tue May 12 11:30:22 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14068] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 455.956 sec [Tue May 12 11:30:27 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 100.275 sec [Tue May 12 11:30:32 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 105.276 sec [Tue May 12 11:30:37 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 110.276 sec [Tue May 12 11:30:40 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14554] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 5 sec [Tue May 12 11:30:42 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @14266] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 115.276 sec

Eric

[jeffkaufman]

Thanks!

Could you paste your LogFormat? Are you using %B to report response sizes?

[eldk]

here it is : LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

[jeffkaufman]

Thanks! So this is bytes sent as reported by %O.

This is really helpful; still debugging.

[eldk]

Here one apache access with good file size : [12/May/2015:16:40:46 +0200] "GET /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg HTTP/1.1" 200 382516 "http://www.google.fr/imgres?imgurl=http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg&imgrefurl=http://www.opalesurfcasting.net/peche_en_surf_casting_mot173.html&h=1704&w=2272&tbnid=SlsuVIADL7HKvM:&zoom=1&tbnh=94&tbnw=125&usg=__fR10g3-pvf-gTfTaQWIi-dcmqzA=&docid=XyKIkjG7qctOwM" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36"

382516 = ~ 382387

xcoucher_soleil_surf_casting_plage_de_vert_bois.jpg.pagespeed.ic.o9Im1AbLI-.jpg, 382387 mai 12 03:43

a warning message from mod_pagespeed (only one - 5 sec) [Tue May 12 16:40:52 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @1696] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg for 5.03 sec

and the same request ten seconds later with bad file size for the same user (same ip)

[12/May/2015:16:40:56 +0200] "GET /peche_en_surf_casting_mot173.html HTTP/1.1" 200 105654 "http://www.google.fr/imgres?imgurl=http://www.opalesurfcasting.net/IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg&imgrefurl=http://www.opalesurfcasting.net/peche_en_surf_casting_mot173.html&h=1704&w=2272&tbnid=SlsuVIADL7HKvM:&zoom=1&tbnh=94&tbnw=125&usg=__fR10g3-pvf-gTfTaQWIi-dcmqzA=&docid=XyKIkjG7qctOwM" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36"

[jeffkaufman]

382516 = ~ 382387

This is expected because %O includes headers

the same request ten seconds later with bad file size

That second request is for /peche_en_surf_casting_mot173.html, not /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg.

but the examples above for /IMG/coucher_soleil_surf_casting_plage_de_vert_bois.jpg are good.

[jeffkaufman]

IPRO has several internal states. If I'm reading your logs right this is happening during the third (serve optimized resource) state.

[jeffkaufman]

It looks like we should be logging errors on failed ap_rwrite calls, though this wouldn't cause the problem.

[eldk]

Sorry for my "bug"

[eldk]

IPRO has several internal states. If I'm reading your logs right this is happening during the third >(serve optimized resource) state.

It's happened in the first state too when first rewrite a non yet rewrited image. For the third state it's look like trying to rewrite thus rewrited image is in cache. This seem to occure more often when rewriting webp images (the images I check are at least 9/10 webp requests) .

Is there no elements in mod_pagespeeds statistic that should help ?

[eldk]

Another case : Image is not rewrited in cache yet. Then one mod_pagespeed thread is launched waiting for completion of url.

Before browser image call : Image in cache (only one jpg) : 528965 mai 12 05:37 xquiberville-plage-est.jpg.pagespeed.ic.o1gRWLsmzT.jpg,

image is asked by browser but no log in apache access log (nothing has been sent yet) : last apache access log for this image : [12/May/2015:05:37:42 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 1418985 "-" "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)" mod_page speed start to send "warning - waiting for completion of URL" : one message every 5 seconds: the last one : [mod_pagespeed 1.9.32.3-4448 @11818] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 411.224 sec Then a log is send to apache access log with status 200 send : [12/May/2015:19:02:08 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 341728 "https://www.google.fr/" "Mozilla/5.0 (Linux; Android 4.4.2; fr-fr; SAMSUNG GT-I9195/I9195XXUCNK1 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/1.5 Chrome/28.0.1500.94 Mobile Safari/537.36"

and one more file is cached (a webp image) : 247995 mai 12 19:09 xquiberville-plage-est.jpg.pagespeed.ic.e7-jnTX07F.webp, 528965 mai 12 05:37 xquiberville-plage-est.jpg.pagespeed.ic.o1gRWLsmzT.jpg,

But the bytes send are not the attended size from cached files or original one : 1418629 déc. 20 2012 /IMG/quiberville-plage-est.jpg

Note the user agent is still from a webp able browser.

This is on First state (with loadfromfile so no http fetch).

Greatings,

Eric

[jeffkaufman]

Testing now, I see http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg as 330170 bytes consistently, which is compatible with the 341728 you saw in the access log, and doesn't indicate truncation. It's interesting that the one you're seeing in the https cache on disk is 247995, substantially smaller, and I can't see where this would come from.

[eldk]

Hello, It's because a third rewrited image has been added. I have now : 247995 mai 12 19:09 xquiberville-plage-est.jpg.pagespeed.ic.e7-jnTX07F.webp, 528965 mai 12 20:32 xquiberville-plage-est.jpg.pagespeed.ic.o1gRWLsmzT.jpg, 330421 mai 12 20:34 xquiberville-plage-est.jpg.pagespeed.ic.tRs1Xq-Dbj.webp,

I look in access log and error log.

Eric

[eldk]

528965 mai 12 20:32 xquiberville-plage-est.jpg.pagespeed.ic.o1gRWLsmzT.jpg, has been rewrited (modification time ls -al) because I disallow a folder from mod_pagespeed and restart apache. (I don't understand why images are rewritted as soon as apache is restarted)

[eldk]

I think those are your access log : 24 - - [12/May/2015:20:32:44 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 383720 "-" "curl/7.35.0" 24 - - [12/May/2015:20:32:57 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 529038 "-" "curl/7.35.0" 24 - - [12/May/2015:20:33:50 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 1418929 "-" "Chrome/100" 24 - - [12/May/2015:20:34:19 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 330494 "-" "Chrome/100" 24 - - [12/May/2015:20:34:43 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 330494 "-" "Chrome/100" 24 - - [12/May/2015:20:35:06 +0200] "GET /IMG/quiberville-plage-est.jpg HTTP/1.1" 200 330494 "-" "Chrome/100"

I remove the last one for url parameters

Look now in error.log

[eldk]

They are there : [Tue May 12 20:32:49 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17834] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 5.017 sec [Tue May 12 20:33:02 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17805] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 5 sec [Tue May 12 20:33:07 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17805] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 10.006 sec [Tue May 12 20:33:12 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17805] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 15.012 sec [Tue May 12 20:33:17 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17805] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 20.012 sec [Tue May 12 20:33:22 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17805] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 25.012 sec [Tue May 12 20:33:55 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17941] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 5 sec [Tue May 12 20:34:00 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17941] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 10.018 sec [Tue May 12 20:34:05 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17941] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 15.018 sec [Tue May 12 20:34:24 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17564] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 5.001 sec [Tue May 12 20:34:48 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17564] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 5 sec [Tue May 12 20:35:11 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @17957] Waiting for completion of URL http://www.opalesurfcasting.net/IMG/quiberville-plage-est.jpg for 5.009 sec

Thread 17941 seems to be the one that create the biggest webp (330421). Then it should be served to thread 17564 and 17957.

[oschaaf]

@jeffkaufman shot in the dark: https://github.com/pagespeed/mod_pagespeed/blob/master/pagespeed/apache/mod_instaweb.cc#L759 ignores a return value that is being used meaningfully: https://github.com/pagespeed/mod_pagespeed/blob/master/pagespeed/system/in_place_resource_recorder.cc#L115

[oschaaf]

@jeffkaufman please ignore my comment, that code path looks fine to me

[eldk]

Hello,

I have been able to reproduce it : bad file size served, long "waiting for completion of URL", and image well rewrited in cache.

I go further to validate checks :

• with no image cached for the type to be served accordingly to the browser request header
• with image yet in cache for the type to be served accordingly to the browser request header

Greatings,

Eric

[jmarantz]

This is great that you've been able to reproduce that. Does that mean that 100% of the time you fetch a particular file from your server, a new thread gets trapped in a "waiting for completion" loop?

I know you said earlier that you mailed your config to Jeff, but I don't have it. Can you specify for me?

1. What other Apache modules do you have enabled?
2. Are you using memcached? If so, are there other servers (e.g. haproxy) between apache and the memcached?
3. If you are not using memcached, what are you using as the file system for your file-cache? Is it a local hard disk? Or something network-mounted?
4. Do you have LoadFromFile enabled?

Thanks, -Josh

On Tue, May 12, 2015 at 7:46 PM, eldk notifications@github.com wrote:

Hello,

I have been able to reproduce it : bad file size, long "waiting for completion of URL", and image well rewrited in cache.

I go further to validate checks :

• with no image cached for the type to be served accordingly to the browser request header
• with image yet in cache for the type to be served accordingly to the browser request header

Greatings,

Eric

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/mod_pagespeed/issues/1048#issuecomment-101461506 .

[eldk]

Hello,

This is great that you've been able to reproduce that. Does that mean that 100% of the time you fetch a particular file from your server, a new thread gets trapped in a "waiting for completion" loop?

Yes. But only for those which are producing a great number of "waiting completion for url" >2 messages and > and completion of url >10 seconds.

1 - I send you and @jeffkaufman the list of apache modules used now (was send to @jeffkaufman, can't find your mail @jmarantz ). 2 - no, I'm not using memcached 3 - The file system cache is on local hard disk 4 - The first error messages occur with httpfetch of original image, but I've moved to LoadFromFile applied only to main image folder now. I think that's this suppress one cycle to rendering rewrited image ?

Try now to reproduce it again.

Thanks,

[eldk]

Ok,

1 - rm all rewrited files from cache for one image 2 - open chrome dev tools with cache disable and modify bandwidth in adaptive view to a slow connection (I use GPRS 50kbps 500ms) 3 - check if nothing has been rewrited for the image choosen 4 - launch load in Chrome with choosen image URL 5 - cancel the load

a - mod_pagespeed start to give warnings in error log " waiting for completion of URL for " a lot as load is slow. b - file size reported in apache access log is the amount of bytes send until the load was canceled. c - I can't stat how to evaluate how fast it is for the moment, but the rewrited is well done each time (less than a second) and I think the time to do it is not as long as sending image to client browser.

So this is, I think not a bug, but a "missconception" in how are handled (if it can be) the warnings when the sending is long due to client, bandwidth usage or maybe other sources.

So for the warning messages that are less than 2 messages and are 1 or 2 warning when sending image to client (the numerous "warning waiting for completion") maybe have the same sources :

• bandwidth ability and usage to send the image to the client
• bandwidth ability and usage from the client to load image

Should you check ?

Greatings,

Eric

[jeffkaufman]

@eldk: Here's a hypothesis:

• The "waiting for completion of url" message means that there's a request that's still in progress
• You're simulating low bandwidth with "GPRS 50kbps 500ms" and your server also has low bandwidth normally
• So large transfers take a long time, more than the 5s trigger for the "waiting for completion" message
• The file size is low when the browser cancels the slow load and you have a partial transfer.

This I should be able to reproduce locally, and I'll see if that's really how our system acts.

[eldk]

Hello,

The "waiting for completion of url" message means that there's a request that's still in progress So, but if a slow client is pushing a request : numerous warnings are produced. Thus there is not really something to warn (except if the rewrite image process is not well closed).

The numerous 5 s one message only and sometimes 10 s second messages happened also when bandwidth output usage is low , for my case between 1 am and 5 am (GMT+2) -

So I think maybe it's a "false warning" message or some process - before sending image - is not closed at the proper time.

rewrite -> send not rewrite->send errors before send then errors after rewrite|not rewrite

Thanks,

Eric

[jeffkaufman]

@eldk When I try to reproduce this locally with chrome set to very low bandwidth I see the pagespeed code finishing very quickly, and then apache slowly sending out the bytes. I need to figure out what's different about your situation that seems to make the writing back up into PageSpeed.

[jeffkaufman]

Same non-reproducing behavior with curl:

curl -o /dev/null http://localhost:8080/mod_pagespeed_example/images/Puzzle.jpg --limit-rate 5000

It downloads slowly, but it's all handled by Apache after PageSpeed.

[jeffkaufman]

I think to test this I need to figure out how to get Apache into a mode where instaweb_in_place_filter will see multiple buckets. Right now even large files (241k) seem to come in as a single bucket.

[jmarantz]

Maybe generate the file with php script that flushes? On May 13, 2015 8:56 AM, "Jeff Kaufman" notifications@github.com wrote:

I think to test this I need to figure out how to get Apache into a mode where instaweb_in_place_filter will see multiple buckets. Right now even large files (241k) seem to come in as a single bucket.

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/mod_pagespeed/issues/1048#issuecomment-101654144 .