search

apache / incubator-pagespeed-mod

Apache module for rewriting web pages to reduce latency and bandwidth.
http://modpagespeed.com
Apache License 2.0
696 stars 158 forks source link

Missing domains from insert_dns_prefetch #654

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
See server2.aura.travel.  Its head is below.  Note that we don't prefetch 
http://assets.aura.travel or http://maps.googleapis.com.  Only 'pinterest'.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head><script type="text/javascript">var 
_gaq=_gaq||[];_gaq.push(['_setSiteSpeedSampleRate',100]);</script>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="language" content="en"/>
<meta http-equiv="PICS-Label" content='(PICS-1.1 
"http://www.classify.org/safesurf/" L gen true for "http://www.aura.travel" r 
(SS~~000 6))'/>
<meta property="og:type" content="website"/>
<meta property="og:image" content="http://assets.aura.travel/aura-logo.png"/>
<meta property="og:site_name" content="AURA Travel"/>
<meta property="fb:page_id" content="220537617995456"/>
<meta property="fb:app_id" content="5622627207"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:site" content="@AuraTravel"/>
<meta name="twitter:image" content="http://assets.aura.travel/aura-logo.png"/>
<meta name="wot-verification" content="c59b16378e027797d7ec"/>
<meta name="norton-safeweb-site-verification" 
content="ccg8yzaokto42jvp20-0bnjsaktj0zl9ds4uty6mc91j3y35fcux5mzwfff058sxbidevuc
vvexifajgh4zbpyjw40y3wx8agd8c8-9b-1jtvk6omy9s7hsaftgfid-q"/>
<meta property="og:title" content="Find accommodation, houses for rent, holiday 
properties and short term furnished apartments"/>
<meta property="og:description" content="AURA is a premier portal for finding 
accommodation in Australia. Holiday houses, rental properties, short-term 
furnished apartments and travel destinations."/>
<meta property="og:url" content="http://server2.aura.travel"/>
<meta name="twitter:description" content="AURA is a premier portal for finding 
accommodation in Australia. Holiday houses, rental properties, short-term 
furnished apartments and travel destinations."/>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<meta name="description" content="AURA is a premier portal for finding 
accommodation in Australia. Holiday houses, rental properties, short-term 
furnished apartments and travel destinations."/>
<meta name="keywords" content="accommodation, rental, rent, holiday 
accommodation, houses for rent, holiday houses, B&amp;amp;B, bed and breakfast, 
short-term rental, short stayz, furnished apartments, properties for rent, 
vacation accommodation, apartments, apartments for rent, holiday apartments, 
villas, cabins, lodges, camping, hotels, hotel accommodation, stays"/>
<link rel="stylesheet" type="text/css" 
href="http://assets.aura.travel/assets/A.fd9c65e7,,_css,,_bootstrap.min.css+fd9c
65e7,,_css,,_bootstrap-responsive.min.css+fd9c65e7,,_css,,_bootstrap-yii.css+fd9
c65e7,,_css,,_jquery-ui-bootstrap.css+e4f76d69,,_bootstrap-docs.css+77a6a857,,_f
ont-awesome-ie7.min.css+77a6a857,,_font-awesome.min.css+d6dd9b98,,_custom-main.c
ss,Mcc.DXBF9xQ_b7.css.pagespeed.cf.MbfW13o3L2.css"/>
<script type="text/javascript" 
src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script 
src="http://assets.aura.travel/assets,_86d623d5,_jquery.yii.js,Mjm._dpwXtgwch.js
+js,_jquery.ui.touch-punch.min.js,Mjm.oXJhGT49Yl.js+assets,_fd9c65e7,_js,_bootst
rap.bootbox.min.js,Mjm.Hd95Bjs8Y4.js.pagespeed.jc.svj8_ynu4X.js"></script><scrip
t>eval(mod_pagespeed_JhmJm64Dbw);</script>
<script type="text/javascript" 
src="//code.jquery.com/jquery-migrate-1.1.1.js"></script>
<script type="text/javascript" 
src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/jquery-ui.min.js"></script>
<script>eval(mod_pagespeed__Z43l41fat);</script>
<script>eval(mod_pagespeed_9As6Zdk0b3);</script>
<script type="text/javascript" 
src="http://assets.aura.travel/assets/fd9c65e7/js/bootstrap.min.js.pagespeed.jm.
-umip9MeSa.js"></script>
<script type="text/javascript" 
src="http://maps.googleapis.com/maps/api/js?v=3.exp&amp;sensor=false"></script>
<script type="text/javascript">//<![CDATA[
(function(c,q){var 
m="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///ywAAAAAAQABAAACAUwAOw==";c.fn.
imagesLoaded=function(f){function n(){var 
b=c(j),a=c(h);d&&(h.length?d.reject(e,b,a):d.resolve(e));c.isFunction(f)&&f.call
(g,e,b,a)}function p(b){k(b.target,"error"===b.type)}function 
k(b,a){b.src===m||-1!==c.inArray(b,l)||(l.push(b),a?h.push(b):j.push(b),c.data(b
,"imagesLoaded",{isBroken:a,src:b.src}),r&&d.notifyWith(c(b),[a,e,c(j),c(h)]),e.
length===l.length&&(setTimeout(n),e.unbind(".imagesLoaded",p)))}var 
g=this,d=c.isFunction(c.Deferred)?c.Deferred():0,r=c.isFunction(d.notify),e=g.fi
nd("img").add(g.filter("img")),l=[],j=[],h=[];c.isPlainObject(f)&&c.each(f,funct
ion(b,a){if("callback"===b)f=a;else 
if(d)d[b](a)});e.length?e.bind("load.imagesLoaded 
error.imagesLoaded",p).each(function(b,a){var 
d=a.src,e=c.data(a,"imagesLoaded");if(e&&e.src===d)k(a,e.isBroken);else 
if(a.complete&&a.naturalWidth!==q)k(a,0===a.naturalWidth||0===a.naturalHeight);e
lse if(a.readyState||a.complete)a.src=m,a.src=d}):n();return 
d?d.promise(g):g}})(jQuery);
//]]></script>
<script type="text/javascript" 
src="http://assets.aura.travel/js/jquery.masonry.min.js.pagespeed.jm.EvS14Bhmh2.
js"></script>
<script type="text/javascript">//<![CDATA[
$(function(){$(".property-list-view .items 
.row-fluid:last").css("border-bottom","none");});
//]]></script>
<title>Find accommodation, houses for rent, holiday properties and short term 
furnished apartments</title>
<link rel="dns-prefetch" href="//assets.pinterest.com"></head>
<body>

Original issue reported on code.google.com by jmara...@google.com on 21 Mar 2013 at 12:58

GoogleCodeExporter commented 9 years ago

Original comment by jmara...@google.com on 11 Apr 2013 at 12:52

GoogleCodeExporter commented 9 years ago 
The reason for this happening is that assets.aura.travel and 
maps.googleapis.com are already present in the HEAD (link/script). But 
assets.pinterest.com is only in the BODY in a div.

Currently insert_dns_prefetch filter ignores domains in HEAD and only inserts 
prefetch tags for domains in the BODY. The reasoning for that is:
- Browser lookahead parsing downloads resources and does dns resolutions as 
part of that. Inserting dns prefetch tags is redundant if we are doing it for 
one of those early resources; but it is useful if we can do it for some 
resource which will get downloaded later and the DNS lookup can be done apriori
- We want to make the bytes in the initial CWND useful. So allowing a resource 
to be downloaded is better than sending out a DNS resolution
- HEAD is our heuristic boundary for this usefulness. Before HEAD - not useful; 
after HEAD - useful.

Let me know if this sounds reasonable. If there is a good reason to insert dns 
prefetch tags for HEAD resources we can do it.

Original comment by bharathb...@google.com on 12 Apr 2013 at 12:53

GoogleCodeExporter commented 9 years ago 
What you are saying makes sense provided that the lookahead parsing behavior is 
standard for the resource domain lookups in HEAD(I could not find an RFC or 
standard for this in a brief search).

The second issue re: should the prefetch for the domains in the BODY be at the 
beginning or the end of HEAD? I am still inclined to say "beginning" - and 
speculate that the overall page load time is faster even if the first window of 
usefulness is made less useful by the extra tags. Domain lookups could exceed 
200ms, which is slower than the second window :) but then again... maybe 
someone could do some stats on this.

Cheers,

Luci

Original comment by l...@aura.travel on 13 Apr 2013 at 9:13

GoogleCodeExporter commented 9 years ago 
I don't think the lookahead behaviour is a standard. This was just from our 
discussion of the usefulness of the dns prefetch tags.

Original comment by bharathb...@google.com on 15 Apr 2013 at 7:14

GoogleCodeExporter commented 9 years ago 
Issue 717 has been merged into this issue.

Original comment by jmaes...@google.com on 2 Jun 2013 at 10:29

GoogleCodeExporter commented 9 years ago 
insert_dns_prefetch's lack of coverage was also noticed on aamjanata.com.

Original comment by jmara...@google.com on 3 Oct 2013 at 1:11

GoogleCodeExporter commented 9 years ago 
On aamjanata.com I see that there's a script in the body whose domain is not 
referenced in the head:
    <script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js">
any thoughts on why we don't add a DNS prefetch tag for that?

Also, I wonder if it's worth revisiting the policy in this filter.   DNS 
Prefetch is focused only on loading domains referenced in the body that were 
not already referenced in the head.  But if the head is quite large, then it 
might take several packets to deliver it to the browser, so you could argue 
that an early DNS prefetch would really help even in that case.  But DNS 
prefetch does not try to estimate that.  It just assumes anything in the head 
is early enough.

Original comment by jmara...@google.com on 3 Oct 2013 at 1:19

GoogleCodeExporter commented 9 years ago 
Hmm, it seems to work fine when I tested. Loading 
http://aamjanata.com/?ModPagespeedFilters=insert_dns_prefetch I see we insert 
the following dns-prefetch links.
<link rel="dns-prefetch" href="//pagead2.googlesyndication.com"><link 
rel="dns-prefetch" href="//ajax.googleapis.com"><link rel="dns-prefetch" 
href="//s0.wp.com">

It looks like the filter isn't currently enabled on the site though, as just 
loading aamjanata.com doesn't reveal any prefetch links. This filter does take 
a page refresh to take effect, since we have to scan the whole HTML to collect 
the domains to prefetch and store in the property cache.

Original comment by j...@google.com on 3 Oct 2013 at 3:17

GoogleCodeExporter commented 9 years ago 
Hi,

I'm Vidyut. aamjanata.com is my blog.

The filters currently enabled are:

pagespeed EnableFilters
rewrite_images,insert_dns_prefetch,inline_preview_images,resize_mobile_images,ma
ke_google_analytics_async,extend_cache,lazyload_images,canonicalize_javascript_l
ibraries,rewrite_css,rewrite_javascript,collapse_whitespace;

As you see, insert_dns_prefetch is enabled, and the server is not throwing
any errors about unrecognized filters while restarting. The prefetch tags
are not getting added to the page with this configuration.

http://www.webpagetest.org/result/131003_C6_PKK/

However, I tried your link with the query adding the filter, and it did
work. This might help pinpoint the issue?

http://www.webpagetest.org/result/131003_AC_PN8/

As you see, the gaps in the waterfall are closing with DNS prefetch on, so
the filter is clearly useful when working.
It is strange that adding it as a query is working, but not adding it in
the configuration. I wonder what it is that I'm doing wrong... I'm hoping
it turns out to be some silly configuration goof up. Does the sequence in
which the filters are listed matter? Should I experiment with putting
prefetch last/first to see if that makes it work?

Vidyut

-- 
Vidyut
aamjanata.com
Twitter: @Vidyut
Diaspora: vidyut@ilikefreedom.org
Facebook: http://facebook.com/theVidyut
Phone: Allergic
Telephone: Forget it.
Mobile phone: Forgot

Original comment by wide.aw...@gmail.com on 3 Oct 2013 at 3:47

GoogleCodeExporter commented 9 years ago 
Is it possible that you've turned it on in pagespeed.conf but off in a 
Directory directive or .htaccess file?

Original comment by matterb...@google.com on 3 Oct 2013 at 5:10

GoogleCodeExporter commented 9 years ago 
Ah, there's also the point that adding a query param won't serve cached content.
Perhaps you could try with some different, random parameter? ?cache=busted or 
similar?

Original comment by matterb...@google.com on 3 Oct 2013 at 5:11

GoogleCodeExporter commented 9 years ago 
Nope. There is no Pagespeed turned off on this server at all. I haven't
gotten around to doing it (for some subdomain pages with already optimized
static content).

Vidyut

Original comment by wide.aw...@gmail.com on 3 Oct 2013 at 5:44

GoogleCodeExporter commented 9 years ago 
I am seeing dns-prefetch tags being inserted on aamjanata.com after 
shift+refreshing a few times, but it is inconsistent. I have a suspicion that 
this filter might be interacting poorly with your http cache setup. This filter 
can take a page refresh or two before it starts inserting the tags, since 
mod_pagespeed first needs to scan the html to collect the domains to prefetch. 
I think varnish might have cached one of the first views of the page, which 
won't have prefetch tags inserted yet. Can you try bypassing varnish and see if 
the tags are being inserted consistently or not?

Original comment by j...@google.com on 4 Oct 2013 at 2:46

GoogleCodeExporter commented 9 years ago 
Tags are being inserted inconsistently.

Varnish has been completely purged several times to be on the safe side in
last day or so and the filter has always been on. In theory, Varnish
shouldn't have had a version without prefetch to cache at all, since the
varnish was installed after the pagespeed.

Just to be clear that there is no question of an inefficient cache purge, I
had deleted the varnish cache physically and recreated it in another place
to rule that out for dead certain, yesterday.

Vidyut

Original comment by wide.aw...@gmail.com on 4 Oct 2013 at 4:28

GoogleCodeExporter commented 9 years ago 
There is something strange going on with the filters. For example, if you
add the prefetch as a query on the url, the CDN rewrites don't happen, but
progressive jpegs work perfectly. This is also consistent.
http://www.webpagetest.org/result/131004_DZ_D2D/

The CDNs that ought to have been used are aj1-cdn.appspot.com and
aj2-cdn.appspot.com - you will see them being used without the filter added
as query along with inconsistent progressive jpegs and compressed images
results.

Am going to make the cache size really large to rule out insufficient space
leading to url being served fresh (and not completely written) each time.
Just to see what happens. In a few hours.

Vidyut

Original comment by wide.aw...@gmail.com on 4 Oct 2013 at 4:34

GoogleCodeExporter commented 9 years ago 
Reviewing some old open issues.

Looking at aamjanata.com again, I'm not able to reproduce the flakiness with 
insert-dns-prefetch. It looks consistent now. Did you find the situation 
improved after increasing the cache size?

Please reopen if you still feel there are any issues with this filter.

Original comment by j...@google.com on 14 Apr 2014 at 3:22

GoogleCodeExporter commented 9 years ago 
Yes, that problem was solved immediately. I think someone kept it because
it needed to be handled better anyway - or something. Don't recall.

-- 
Vidyut
aamjanata.com
Twitter: @Vidyut
Diaspora: vidyut@ilikefreedom.org
Facebook: http://facebook.com/theVidyut
Phone: Allergic
Telephone: Forget it.
Mobile phone: Forgot

Original comment by wide.aw...@gmail.com on 14 Apr 2014 at 3:40