search

arvgta / ajaxify

Ajaxify - The Ajax Plugin
https://4nf.org/
274 stars 124 forks source link

traffic / load reduction #30

Closed wikiloops closed 9 years ago

wikiloops commented 9 years ago

Hey, Just spent the whole day looking at your fine piece of code, but besides the due praise, I'd like to point out a few things I noticed which might be relevant to some others as well: Obviously, ajaxify is working really nicely in a context with static pages that are not likely to be changed during a visitors stay, since I am working in a much more dynamic context, the whole idea of prefetching and storing content is of no use. Since every request to a database driven page will cause load, I am most interested to only trigger such requests when the user actually needs them - especially when it comes to pages like search results that cause a huge database load to get rendered. After disabling the prefetch & cache options of ajaxify and working out some project related issues, I noticed ajaxify still issues a second request of the page it is initialized on (http://4nf.org does, too). I am not sure I have exactly understood why this is needed, but by disabling the request on line 254 of ajaxify.js, this does no longer occur, while all expected functionality remains.

I'd recommend disabling this call along with the prefetch-disabling, such double-requests would defintely be problematic in my use case, where a lot of "landing" visitors hit quite load intensive pages.

Thanks for your efforts!

arvgta commented 9 years ago

Hi Wikiloops!

Here, we're talking about this line of code:

$.getPage(location.href, $.scripts);

I tried disabling it by commenting it out and that causes hard errors on http://4nf.org/ .

The above line is necessary for initiating script handling (delta-loading).

As far as changing the defaults for prefetching and memory is concerned, I'd ask other users to comment as well.

It might be, that due to the highly dynamic content of your page, that you have different demands...

Cheers!

wikiloops commented 9 years ago

Looking forward to others opinions :) I see a very great potential in investigating this kind of implementation. I was initially a bit irritated by the statement about ajaxify saving traffic, which it does not in its current setup - it does not save any webtraffic, it just parses less of what it receives than a standard page load would. For those with access to their php code, it is very simple to really save some traffic by wrapping parts of their page in a function that will actually reduce the number of bytes to transfer from server to browser by only sending the "#content"-div and a reduced header.

for example:

<?php
///detect if page is requested by ajax 
$ajaxifycall=false;///standard case
if (!empty($_SERVER['HTTP_X_REQUESTED_WITH']) &&
strtolower($_SERVER['HTTP_X_REQUESTED_WITH']) == 'xmlhttprequest') {

/* one might also want to check for referral header to make sure 
its an internal request from ones own page here  */

$ajaxifycall=true;
}

////simplified page structure
<head></head>
<body>
if ($ajaxifycall==false) { 
      <div id='somestaticstuff'></div>
                                }
      <div id='content'></div>

if ($ajaxifycall==false) { 
      <div id='somemorestaticstuff'></div>
                                }
</body>
?>
arvgta commented 9 years ago

Hi Wikiloops!

Thanks for your extensive reply and the idea, how to reduce traffic. First of all, Ajaxify claims to "avoid full round trips and creating faster page loads", afaik. Where did you read "saving bandwidth"? I'd be ready to correct that immediately, because that is obviously not the case, with or without "prefetch".

The technique to reduce traffic you're illustrating above is the originary approach of "Pronto". It results in genuinely reduced bandwidth and even faster page loads, but requires some server-side customisation, as you described above. Also, I reckon, with the above approach, as you've illustrated, you'd even have to set

deltas

to "false".

Now, "delta loading" is an advent, that most users enjoy! Without it, you'd have to somehow handle the corresponding logic on the server-side. What also comes to my mind is that the "memory effect" does genuinely reduce traffic and dramatically improve page load speed...

In contrast to the typical "Pronto" approach, Ajaxify aims to have rapid time-to-market and minimal customisation. Its strategy is to turn a conventional site into an ajaxified one "out of the box".

It is also very important to handle users that have Javascipt disabled. I trust you'd detect that and provide a second variant of the pages?

That obviously means double labour, not to mention a double code base.

So it's a tradeoff:

1) Time-to-market + less labour (the original Ajaxify approach) 2) Faster machine response time + less bandwidth, however with much more labour + impaired time-to-market

Thanks very much!

wikiloops commented 9 years ago

Hey :)

You are right about disabling delta loading, I used to use the balupton version of ajaxify which didnt feature delta loading (or at least I didnt come across that), so the trouble of working around scripts loaded by ajax has already been taken care of in my case. Once I got used to the special behaviourisms depending on the positioning in- or outside the pulled content, and I really like the reduction of bandwidth/single requests that goes along with that.

You are absolutely right about the tradeoff situation here, and I was by no means criticising the out-of-the-box approach you are taking. I was rather thinking that it would widen the use spectrum of ajaxify if one would offer another option which could be set to "false" if not overridden, and which would enable the modus I am using in one go - no memory, no delta, no prefetching, but an extra header or something sent. Such an option plus a copy paste php snippet like the one outlined above might be interesting to more people for the given reasons. Implementation is not that much labor, once you have figured out how keep your script functional by using $('#content').on('handler','click',function(){'do it'}); instead of $('#handler').click(function() { do it } ); what you are suddenly looking at is something that behaves pretty app-like once the first page is loaded. I'd definetly say this is interesting, especially thinking of mobile use cases where every server roundtrip counts.

There is no need for the extra checkup for browsers with JS disabled, simply because a browser with JS disabled will not send the 'HTTP_X_REQUESTED_WITH' header, at least none I have tested so far. If there is no such header, the whole page is served, so there is no need for a duplicate codebase.

Since your version is one of the better documented ones, I felt it would be nice to add this info to your project, even tho it may be usefull for geeks only :)

all the best!

arvgta commented 9 years ago

Hey there,

yes, that's very important, that the plugin gracefully degrades in this case, too. Very convincing what you're illustrating - it seems like an elegant variation, as you say more for geeks.

What kind of "change request" are you placing then in a nutshell?

As far as I've understood, you would like an additional plugin option, probably to send something more specific than the HTTP_X_REQUESTED_WITH header.

I'll be happy to supply it in the coming version of Ajaxify - yes! Would you like it in addition to the HTTP_X_REQUESTED_WITH header, or even instead?

Thanks for your sharp thinking!

I might illustrate your approach on http://4nf.org/ on the PHP-side as well, this issue kindly serving as documentation in the meantime...

So please just let me know, which enhancement on Ajaxify's side you need...

Thanks!

arvgta commented 9 years ago

Hey Wikiloops!

Another thing came to my mind in the meantime - I could modify the line that produces the second GET to:

if(deltas) $.getPage(location.href, $.scripts);

i.e. NOT perform the $.getPage() if deltas are disabled.

You have already tested that - so I'll enable it by default @4nf.org and @github!

Concerning turning

prefetch

default to false, please have a look at these demos

Only 3 out of 10 have "turbo/prefetch" turned off...

wikiloops commented 9 years ago

Hello again :)

I have a lot of things on my plate right now, so please excuse me for being a little slow in reply these days. First of all, I am glad my explanation of concept seemed to make sense to you :) I will need some more time to identify the crucial needs of my approach, so I can define exactly what the new enhancement should include.

Your suggested switch to 

if(deltas) $.getPage(location.href, $.scripts);

makes total sense to me, as far as the prefetch is concerned... well, let me try to put it this way: Since prefetch can be turned off by the current version, maybe there is no need for a seperate handler any more, except for the special header, which I'll return to in a bit. After looking at the pages on the demo site, I couldn't help the impressions that those I looked at propably do not have harsh loads of traffic (>1k visitors a day), I may be wrong about this, but maybe they just do not feel the nerdy need to save extra hits... If I imagine someone operating a database driven shop which offers long lists of item categories, or links to filter down by certain criteria (think of ebay as an example), a slowish mouseover move of the user who seeks for the correct list entry will cause as many prefetch hits as he bypasses items. When the user finally finds the wanted item and clicks on it, chances of having a huge load time experience because all of the prefetch traffic going on are pretty high I'd say. Lets not forget the server doesnt realise its "just a prefetch hit" and will go to work sorting and serving on every prefetch hit, regardless of the users final decision to click or not to. Of course this will only happen once if memory mode is active, if it isnt for some reason, this is a dangerous scenario.

So, now back to the special header issue: Maybe there is no need for action here either,

I have too little knowledge about the concept behind this header and what other functions may rely on its presence, so I wouldn't disable that, either.

So, if you'd still want to add something that distinguishes a call from ajaxify from any other call, you could either integrate a seperate header OR attach a variable to the called URI which could then be caught by some

$ajaxified=false;
if ( isset($_GET['specialajaxyvar'])) {
$ajaxified=true;
}

This might be the even simpler approach, for most people will be more common to working with $_GET parameters than with analysing headers. I'm not sure if sending custom headers is considered good practise, I just dont know really.

At this point, the enhancement we are talking about might be setup like this:

jQuery(‘#content’).ajaxify({
nerdmode: false; 
/* will disable delta & prefetch & send an additional identifier 
as defined by "nerdkey" and "nerdkey-type" if set to true    */

nerdkey-type: 'none'; 
/* needed in nerdmode only: possible choices: 'get', 'header' or none  */ 

nerdkey:'none'; 
/* needed in nerdmode only: define header/get parameter name, value will always be 1 */
});

This would allow a very flexible choice between the two identifying options, if then there is really need for them. The idea to let users define the name of the header / get variable is a nice extra, of course a fixed key would work just the same if its uncommon enough.

Theres some more minor things and another approach on how to integrate adsense I am thinking about... I'll give you more info as soon as I have clarified if my approach is worth sharing here.

Oh, and last but not least: some commenting about /// may be removed if using nerdmode in the ajaxify.js script would be very nice :) I assume I could do with a much more leightweight script since I have disabled so many of its features, and i have not dared to remove any of the functions fearing I could mess up the whole script. I'm not that much of an JS expert as you may have noticed.

arvgta commented 9 years ago

Hey :)

Wow! Thanks for the extensive reply!

Just went to a client website of Ajaxify, probably the most prominent:

So Ajaxify seems to be fit for these kind of sites as well. Of course the goal is to cater for a wide as possible spectrum of sites - from novice developer to geek. Of course, a site like eBay is currently out of scope...

As you may have guessed, the site above has turned off "turbo"(now called "prefetch"). They have also turned off "memory". However they have delta-loading turned on.

The thing about turning off "prefetch" from the beginning is that "preview" is also impossible then (i.e. currently "preview" relies on "prefetch" to be turned on).

Also, turning off lot's of options obviously reduces "what Ajaxify does for you" in the first place. Geeks will be comfortable with that, whereas the relief/aha-effect for beginners may be less.

The idea is to get an Ajax-ready site up und running afap (living prototype principle). Of course, the developers can then fine-tune their Ajax functionality (progressively). For example, there are facilities to disable "memory" selectively, or "preview" just the same.

I don't mean to be repeating myself again and again, but I do place some things over efficiency in terms of machine time:

Of course, if server load gets too steep, some compromise has to be made. However, high server load has facettes of a good complaint and has to be achieved, first ;)

Now, we're at the point of specifying the last bits of your original change request, most notably the additional header bit or a URL variable.

I would favour a simple additional parameter. You clearly said, you would favourise a URL parameter that can be pulled with a

I would say, soft-coding is appropriate, letting the developer baptise the hard name according to his own taste. What do you think of "pulse" or something cryptive like that?

You mentioned commenting the code more extensively, especially in order to strip-out bits and pieces. I reckon, I can do that for the stable bits of code. Your notion, that you only need maybe half of the code is correct, if you don't even employ delta-loading. I'll get to that progressively, especially in calmer areas of the code. You do know that we have Ajaxify on a JS content delivery platform. That was reported to give a speed improvement on initial load of about 50%.

You mentioned a way of supporting Adsense in the framework. Yes it is unbelievable, that a giant like Google has yet failed to offer a solution for the Ajax audience. Tell you what - if you come up with a feasible design, I would like to reward you with a token of cash! Have you seen this issue yet?

There is also another efficiency issue, that I came across, originally pertaining to 4nf.org, but potentially pertaining to most user sites:

That was criticised by a great tool called vnseo.

Do you reckon, it would be possible to create an "out of the box" solution to that in Ajaxify?

I really appreciate your very positive contributions, despite the shortage of time! Do you have a URL, which I could support with some backlinks or even on the 4nf.org demo page?

wikiloops commented 9 years ago

OK, I'll make this one a quick one: Just looked at vnseo - what you see down there are googles pagespeed testing results, something anyone concerned about their SEO should look at at some point. I have tried to optimize for speed using exactly this tool, and it will always complain if you have so called "render blocking" css or script files. You may gain points by either seeting your external scripts to async mode (watch out for bad user experience if that takes too long), or by moving down the javascript calls as far as anyhow possible in your document, so the above part may be rendered "unblocked". The even more important issue is external CSS, because the browser will lock untill the file has been downloaded. If you have external CSS in your header (where it should be), none of your page may be displayed before that has been loaded. To get really good page speed results, they suggest inlining the CSS so there is no need to wait for the external document. That works, but your pages document will increase in size if you put all the CSS in there, and of course you'll loose the benefits of a cacheable external file. However, if you use ajaxify in the bite-saving mode we have been discussing here, you can prevent the output of that inlined chunk of CSS on the server side. Result: great page speed test results and still minimal transmission when using ajaxy.

About the adsense thing - I see this as one of the major, major downsides of ajaxy in its current state, and I'd suggest following a very conservative route when finding a solution. People are very aware of the strictness of adsense TOS, and will certainly not want to use any shady workaround that might end in their google accounts being closed. What I'm trying to do is similar to the approach I have seen documented here, where the only successful implementation worked by loading the ads outside the content div and placing them by CSS / "position:absolute". I'll post up another thread on that as soon as I can tell if it will really work and not kill the revenue... I understand why adsense will not allow ajax calls, counting ad impressions and making sure people dont load ads to hidden divs over and over again will be quite difficult, so turning it off all the way might be the safest route from their perspective.

arvgta commented 9 years ago

Thanks for all that!

There are quite a few topics open:

1) The new "pulse" option. that you requested - you didn't mention it anymore - is it still deemed useful? Or is the standard Ajax header enough?

2) "Above the fold rendering" - I trust it can't be solved by Ajaxify and you said that the tool(s) always moan about that. So I'll discard that idea... - it's off-topic from the point-of-view of these threads anyway - just came to my mind.

3) Adsense - I agree that this issue is a major turn-off. However, I would like to add, that it pertains to loads of Ajax sites... You said you can understand it from a Google point-of-view. Nevertheless, a top-notch company like Google should be able to offer a solution, I think. If we would arrive at a generic solution within Ajaxify, that would be revolutionary! After all, we have the full intelligence of Ajaxify on our side. Ajaxify "knows the site" quite well and there are means of e.g. ignoring blocks of JavaScript etc. or retriggering them.

This site: Dark Realm Gaming employs Ajaxify along with Adsense. They worked around it by not placing the ads in the content div, just like you said.

On a side-note, here's their Ajaxify call:

$('#content, #foot').ajaxify({
        forms: false,
        inline: true,
});

i.e. they use "prefetch"/"memory" and the likes...

Just found this discussion on Adsense for Ajax.

Please see quote:

"There is actually pretty good support, but you have to use AdSense through Google DFP - https://www.google.com/dfp/. Things like creating ad slots dynamically / loading them later / refreshing them dynamically work fine when you use GPT tags in DFP."

That seems to be quite new, as I've never seen it before and now it's top at Google for "Adsense Ajax"!

wikiloops commented 9 years ago

OK, lets not spread this discussion here too widely :) As for your questions: Yes, I believe the "pulse" option might still be nice - I am not promising I will use it in my own project, but if its possible without too much work, it would ad some really nice customization to ajaxify - so why not have it? Since its obviously a little more complex to use, having a good documentation on the concept of this feature will be crucial. I am willing to help outline the basic ideas so others can benefit from that. The setup I am currently using is (I know you will have tears in your eyes now :) )

 jQuery("#content").ajaxify({
fade:0,
style:false,
forms:false,
prefetch:false,
memoryoff:true,
cb:somefunction,
inline : false,})

Lets not go back to the "does preload /memory etc make sense" debate - it does in some cases, and it doesn't in some others, depending on the projects overall approach.

I have investigated the links you shared, I believe it will make sense to explain my approach towards the ad problem in another thread/issue. I am not promising anything revolutionary, but to move from "ajaxify does not work with adsense" to "if you want to use ajaxify and adsense, we have a setup tutorial here" would be a step on the way :) Please excuse me if it will take some more time, I want to do some real life testing before leading people down a bad path.

arvgta commented 9 years ago

Hey :)

First of all, I've researched the code for appending to the URL transparently. The following seems to be rather cryptive:

foo.href += (/\?/.test(foo.href) ? '&' : '?') + 'pulse=' + escapeURLComponent(pulse);

see source thread

(didn't see a special jQuery method for it)

Yes, your call of Ajaxify above is very minimalistic, but we agree that it's horses for courses. (Didn't mean to be arguing - sorry)

Looking forward to your Adsense approach - best of luck, while testing it. No, we definitely don't want any Adsense users being banned... Take your time!

Yes, illustrating the Adsense + Ajax approach with a small tutorial would be better than what we have now.

Do you agree with the above code for "pulse"?

EDIT: Tried the above approach for "pulse" and it seemed to impair the POST.

Any ideas why? Or should this be disabled for POSTs? Also, it adds quite a bit of weight to the code...

Cheers and thanks ever so much!

wikiloops commented 9 years ago

Hey - i think we can have it a bit easier than your solution, looks like we had a misunderstanding on the part i referred to as "nerdkey": In your example, you are attaching "pulse=CUSTOM_VALUE", i was thinking to send "CUSTOM_PARAMETER=1". There really is no need to send a custom value, for users will only test for the presence of "pulse" to identify an ajaxify call, it doenst really matter which value is attached. My idea to let people chose a custom parameter was intended to make sure no one who may for some reason have a parameter "pulse" in his project runs into trouble because it now appears twice - I imagine if you called the parameter "start", quite a lot of people would have conflicts between their pagination parameter and ajaxify, thats what I was wanting to prevent.

As for the POST problem - I have not fully understood what the issue is here. Ideally, the additional parameter should still be attached in form of a GET parameter, please dont try including it to the POST parameters, otherwise one would have to check for $_POST['pulse] AND $_GET['pulse'] on server side all the time. In my use case, most POSTs will be followed by a redirection to a page displaying what has been entered via POST, so the POST-receiving script will not send much more than a 

redirect('post_success_page');

there is not many bites to save in this, so I wouldn't mind if the parameter wasnt attached to such calls.

I am absolutely happy with the way you are handling my rather odd requests here, and I know how hard it is to switch concepts of perception. My approach is leading in a very diffrent direction than what you initially intended, so I am aware it will be my job to supply the use cases and show exemplary benefits. I'll do my best :)

To answer another open question from this thread - you were asking where I read ajaxify was saving traffic... I came across the statement on http://4nf.org : "avoiding a full page refresh and the associated full round-trip and creating awesome page load time." - i believe it was the term round trip that confused me a bit, for its a term I only know in traffic related context. The call & response from browsers request header to returned server reaction is taking place, so the roundtrip is not avoided IMO.

I have set up a test version using my current approach at http://www.dev.wikiloops.com last night (will be gone in a few days), feel free to have a look! I'd like to offer some insight in the pages requirements, which I hope will make it easier to understand the needs I am proposing. Thing is, any page you visit there must be generated dynamicly, because there is

So, I was serving one out of six diffrent versions of each page before starting with ajaxy. When using pulse-mode to reduce the number of bites to transfer, those six variants again double to provide diffrent servings, dependent on the standard load vs. ajaxy load distinction we have introduced here. This may sound extremely complex to do from one single codebase, it all comes down to wrapping functions in logical if statements to enable or disable them according to the visitors members-, mobile- and ajaxy-status, which is easy to do in .php. At least a lot easier than creating and maintaining twelve codebases.

I have spent an extensive amount of time optimizing speed test results as provided by https://developers.google.com/speed/pagespeed/insights/ and YSlow, and ended up in a trade-off situation of good page speed result by inlining CSS and JS, resulting in large page-bite-sizes = a lot of traffic vs. mediocre page speed results with "render blocking" external CSS an JS, benefitting from cacheable external files and less traffic per page view.

This is what the "pulse" mode will solve: By placing all inlined CSS and JS in the "calm zone" outside of the "#content"-div, and serving only the content div when a request is called via ajaxy, the downside of the good-page-speed-variant can be levelled out. The CSS which is kept outside the #content will be automaticly applied to pages loaded into it by via ajaxy. As for the JS which shall be parked in the "calm zone", I needed to do some changes so my functions would recognize the presence of new content loaded by ajaxy, mostly resolving to the 

$('#content').on('handler','click',function(){'do it'});

format using jQuery on(). Not that difficult, either, and it resolves the need for any "fake" document.ready calls after any ajax page-load.

_So, as a first conclusion about pulse-mode: _ by re-ordering your code, and changing any jQuery mouseover or click event listeners to $('#content').on(), you may improve your page speed test results without having to serve the same CSS and JS to the same visitor over and over again. Needless to say good page speed results will be great for your SEO as well.

As a last insight, I'd like to point out my personal likings of what I called the "calm zone" (= anything outside of #content div). People might think that this is the "dead area" of a webpage, but thats a wrong perception. By using ajaxify, it is very easy to do nice things that just stick to the browser while the main content is loaded - think of small facebook-conversation windows and Facebooks right-side "live" information panel which both are kept in shape & position while you are browsing around. Guess what, you can do just that by placing stuff in your "calm zone". If you are familiar with absolute positioning, you can even let objects from the calm zone appear to be inside (or on top of) the visual area filled by the #content div. Just because those objects remain static as you browse the ajaxified content, that does not say you cannot have functionality within those objects, the "calm zone" may hold just as many active & interactive things as the content. On wikiloops, the #content div may be used to browse from track to track, while the group & 1-on-1 chat windows offered to members stay calmly where they are as the content fades in and out - thats not only good user experience and visually nice, it also saves a lot of bandwidth at the same time.

Sorry for the extensive read, I felt it might be of help to understand why I do what I do, and it will help a lot when looking at the adsense implementation later. Dont be surprised to find not all speed relevant issues (like inline CSS) are at the works on my current beta, it is still under development :)

arvgta commented 9 years ago

Hey :)

Absolutely amazing! Thanks very much, once more!! You obviously have more experience in the front-end, as I used to be a back-end expert. I'll progressively answer again here, as you mentioned so many points:

Just a short briefing to let you know, that I've read your contribution and fully agree...

1) The "nerdkey" issue - solution/interface to the developer:

nerdkey : "pulse"

and only apply it to GET's - which solves the funny, I was experiencing with the POST. You had mentioned already, that it should apply to GETs only.

I'll send this:

?pulse=1

with the following code from this thread:

foo.href += (/\?/.test(foo.href) ? '&' : '?') + 'pulse=' + escapeURLComponent(pulse);

I'll do that first thing in the morning :)

arvgta commented 9 years ago

Done, with default:

nerdkey : false

Test file

I noticed that, when it is enabled, it greatly slows down the rendering?

Please test!

Read your very extensive illustration in the new thread (#33). Don't want to interfere with my comments :)

Thanks very much!

wikiloops commented 9 years ago

just had a look at the updated file - I have no idea why this small addition should take a long time, really, but i have not tested it yet. One operation you may save is the encodeURIComponent('=1') bit - since there is nothing to encode about '=1', I'd just drop that.

about #33 - I'd be happy about any comments, I'll ad some more thoughts on the issue as I go - if theres something crucially bad about my approach, feel free to adress that, I don't claim to be a javascript expert at all, so I'm really happy to get some feedback :)

arvgta commented 9 years ago

Done, with the following code:

if(nerdkey) hin += (hin.iO('?') ? '&' : '?') + nerdkey + '=1';

I've double checked the speed isssue: If nerdkey is enabled the rendering is very slow on http://4nf.org/ ?? (keep in mind, that I'm using all features including "preview" and "squeeze")

Yes, I'll comment on the Adsense thread asap. But please do test, whether the new "nerdkey" is performing for you...

I've entered a $.log message that dumps out any Nerdkeys generated. That looks fine, only one Nerdkey being generated per GET. However, when

squeeze: 200

is enabled as well, the Nerdkey is generated 2 - 5 times - when I enter something in the search form!

If I disable all page transitions, business as usual (only one Nerdkey generated), but still slow.

So, in a nutshell:

I've left the combination

squeeze: 200,
nerdkey: "pulse",
previewoff: false

plus the $.log()...

up and running at the moment at http://4nf.org/

If it's alright for you, I'd like to debug this first, before adding more features? (I suspect two independent bugs)

Thanks and I might be off for a couple of hours now, but intend to reply to the other thread today...

wikiloops commented 9 years ago

sorry, one more: it might be the if (nerdkey) part as well - this will work if nerdkey is true or false, as soon as its set so a custom vairable like e.g. "pulse", I imagine the test will have to guess wether that is supposed to be true or false... maybe try if (nerdkey != false) instead? Maybe the "canonical" option notices the new parameter (and its lack of presence in the page canonical link), too, and triggers a second go around? One might have to exclude the "nerdkey" parameter from the canon-handling then... Just troubleshooting a little...

Since i am working in a modified & compressed js version, it will be a bit difficult for me to test your updated version, will do as soon as possible tho. Have i missed any bigger changes besides the added code we have been looking at here?

arvgta commented 9 years ago

I've modified the code in Ajaxify to:

function _lAjax(hin, p, post, pre) { //execute Ajax load
  var nurl = hin;
  if(!post && nerdkey !== false) { 
     nurl =_addNerdKey(hin);
     $.log("Nerdkey added : " + nurl);
  }

  var xhr = $.ajax({
    url: nurl,
    type: post ? "POST" : "GET",
    data: post ? post.data : null,
    success: function (h) {
      if (!h || !_isHtml(xhr)) {
        if (!pre) location.href = hin;
      }
      $.cache($(_parseHTML(h)));
      $.pages([hin, $.cache()]);
      if(p) p();
  },
  error: function(jqXHR, status, error) {...
  }
  });
}

Same behaviour, though. So what I'm doing is passing the Nerdkey to the Ajax request only. The buggy behaviour is only, in the event of a GET, from the search form... Just saw, that the search form on http://4nf.org/ employs a GET - NOT a POST!

Here's a minified version of the current Ajaxify if that helps...

No, there are no more changes in Ajaxify than the ones we added :)

arvgta commented 9 years ago

Regarding this issue:

avoiding a full page refresh and the associated full round-trip and creating awesome page load time

I believe it is true, at least in the case of absolute URLs. These cause a full-fledged DNS lookup, right?

Performance of these lookups may have improved, but I believe the underlying architecture is still the same?

wikiloops commented 9 years ago

oh, sorry, i missed you updated one of the above posts with some additional testing info!

One thing is for sure - nerdkey can in no way slow the server down - all it does is send a GET parameter which will simply be ignored at server side, unless you check for it and have some extra functions happen. Please lets not start a myth here :) Maybe the attached parameter prevents some of the memory caching, which would explain a slower user experience, but whatever happens, it does happen on the browser side o things.

I'm affraid there are some bugs in the form handling and potentially the squeeze effect (hey, if it issues 5 nerkey logs, it propably posts/gets 5 times!! Some people call that a DOS attack :P ) There are actually quite a lot of visual bugs on 4nf.org and I have run into trouble navigating by firefox on a regular basis (p.e., the head navigation doesnt always react to clicks, and the previews tend to be displayed in narrower and narrower boxes the longer I surf). Please dont take that as offense, I know one developer cant possibly spot it all, and I rely on user feedback just the same :)

bottom line from my point of view - I dont believe the slowdown will happen if you disable what I have disabled, and the whole "pulse" thing only makes sense without p.e. memory enabled, so testing that together is not really what we need. When implementing the out-of-the-box form handling, I ran into quite a lot of cases where it wouldnt work right away. For example, I had the impression form handling gets stuck if the returned uri is identical to the page I submitted the form on - since ajaxify prevents fetching the page you are on. My log-in forms wouldnt visually act at all because of this, even tho a page refresh proved the log in had happened. When trying to work around this by attaching a dummy variable to the page which is returned after a form submission, that again was overriden by the activated "canon" modus, and again no change taking place :) After i made sure the dummy var also appeared in the canonical header, all worked fine, but lets be honest - thats not out-of-the-box. If you feel the whole "pulse" thing is too shaky to implement into ajaxify, you may still inform people of the option to reduce their traffic by checking for xhtmlrequest-header - surely didnt intend to make your life harder here :)

last issue - I am not sure why you believe a request issued by ajax will not need to run a DNS lookup. I am no expert on this, tho. Just because a relative path link does not include the server hostname, I don't think connecting to that page will be possible without resolving the server - I have simply never ever seen a console log showing any files without the servername, no matter what the reference is. I'm no fundamentalist about the need to change that sentence, just wanted to answer your initial question why I was mistaken at first.

arvgta commented 9 years ago

Hey :)

I just did a test, and I'm afraid - it's exactly as you said:

There are several GETs fired after the submission of a form, the new transitions just making those visible, what I otherwise couldn't tell, because it happened so fast...

Test file at the moment

What I've done is reverted the file back to a backup without the nerdkey - to simplify the algorithm a bit, but have left the flexible forms selector active, because I don't think, that's a problem...

You can see a live demo of how the forms business is creating several GETs, depending on how many passes already have been made.

I would like to apologise but really was not aware of this underlying bug.

I reckon, this has to be debugged asap, before adding new features.

Sorry about that, and thanks once more for your keen perception!

(if we can debug this embarrassing bug successfully, I'd be happy to re-introduce "nerdkey")

Tried turning canonical to false or previewoff to true - bug persists... (checked against our partner site: http://www.oeko-fakt.de/ - bug also visible there (left active))

Bug presumably found! - Line 762

_ajaxify_forms();

was presumably being called over and over again. With every content div switch, however globally! (this probably attaches multiple submit handlers to one and the same form )

By commenting it out, it looks a bit better :) .

Now, I have to facilitate a means of re-ajaxifying forms in the switched content div - that's all!

:

wikiloops commented 9 years ago

Take your time to find and eliminate those things, I am just throwing ideas your way, no need to apologize, I'm fine hacking my way about, and firefox console is my brother ;)

Some really nice results to demonstrate the power of the "pulse" idea in correlation with google page speed results:

I tested the mobile start-page of dev.wikiloops.com in four loading /setup versions, here is the result:

Variation1: Standard load with external CSS file

Transferred document: 4.1kb PageSpeed: 78/100

Variation2: ajaxify loading of the very same page in pulse mode

Transferred document: 2.9kb (30% saved)

as opposed to

Variation3: Standard load with inlined CSS as suggested by page speed

Transferred document: 8,2kb PageSpeed 96/100

Variation4: ajaxify load in pulse mode

Transferred document: 2.9kb (55% saved)

The "transferred document"-sizes are not including any assets like pictures or other resources, this is just the html document, including everything on standard load, and reduced to the #content when the page is loaded thru ajaxify. I believe the numbers speak for themselves :) ajaxify combines the benefits of a cacheable CSS and the speed of inlined CSS, while saving you at least 30% of bandwidth from the third page load on.

arvgta commented 9 years ago

Done - test file here

Richard, do you have a form somewhere in the content div so we can test that pathway, too?

In the above speed tests - are you using the nerdkey option already? Should I re-introduce the nerdkey option now? (I would say, after testing - if at all)

Your testcases seem to suggest that the Variation4 is the fastest, I presume? Or is Variation3 faster?

Either way it seems to entail, that Ajaxify is in general good for the page load speed :)

wikiloops commented 9 years ago

Hmm, looks like I need to explain that data a little - the test actually compares TWO variants of using ajaxify, the first one in combination with a standard built page, the second one using the CSS inlining, as suggested by page speed.

So, the increase in page speed score is due to the CSS embedding, not to using ajaxify. That was not unexpected. There are however some reasons not to inline CSS, one of them being the much bigger document size, another the fact that inline CSS will not be cached but must be transferred over and over again. Using ajaxify in the pulse configuration solves those two problems. By preventing the output of the inlined CSS when a page is loaded via ajaxy on the server side, the bloated document size problem is gone. And we can safely exclude the CSS, because we know: If a page is called from ajaxy, the needed CSS is sitting outside the content div now and will be applied without re-sending it. So, the question I was wanting to answer with this test was: how much traffic can be saved by sending only the content. So, variant 1 and 2 will give you a figure for a normal page setup, and 3 & 4 demonstrate the effect of ajaxify pulse is even more interesting if you care to optimize for speed by inlining CSS. This approach is aimed straight at mobile developers, to whom this is of great concern.

Since the pulse mode will in any case not work on the very first page load (="harsh load", variation 1 & 3), we will always look at two transfer sizes, one for the harsh load, and one for the reduced-output that is returned when a page is loaded by ajaxify AND the server is configured accordingly - thats variants 2 & 4, then.

Pagespeeds Test, Bots and users with javascript disabled will never use ajaxify, so they will always transfer the full document - thats why the four cases only feature page-test results with the "harsh load" results. There is no way to always achieve the variant4 results, we are not doing some magical output compression, we are just sending a reduced page when we can be sure all the elements outside the content have been transferred before. How can we be sure? Because the server is configured to distinguish when a page is called thru ajaxify :)

Let me try to bring a little order into the open issues we have here, by summing up a little.

I'll have to see what I can do, currently focusing on the adsense evaluation and some other tasks. Ideally, one would set up a test domain for form testing, using 4nf.org itself is popably not the very best testground, simply because you wouldn't want to destroy the functionality.

arvgta commented 9 years ago

Hey :)

Thanks for elaborating on that!

Just some quick feedback. Alright, so variants 2 & 4, afai understand are the typical ones when using Ajaxify. The drawbacks of inline CSS are lifted by Ajaxify in variant 4, in pulse mode, it being a nice-to-have... So variant 4 is the best of both worlds in terms of speed (for speedoholics), however does not apply to the first "harsh load", which is inevitable...

I would like to re-introduce it, once we've tested the nasty old bug with forms. Do you have a non-critical playground somewhere? Or otherwise I could use the old partner site http://www.oeko-fakt.de/ because it has virtually no traffic..., installing e.g. this WP plugin

(I would also like to push the debugged code without pulse to GitHub before we get going with the new feature...)

wikiloops commented 9 years ago

You wrote: "...so variants 2 & 4, afai understand are the typical ones when using Ajaxify" No, got to disagree on your evaluation again, sorry.

Lets forget about all page speed issues for a moment, those have to do with SEO and optimal performance and will be only of interest to people who are concerned about that.

If you want to draw a conclusion concerning ajaxify.js, the following statement is true: When using ajaxify out-of-the box, the traffic caused to load a non-cacheable page is identical to the traffic caused by a traditional page load - that's the "typical" behaviour when memory is off, which it must be to retrieve fresh dynamic pages. When using ajaxify in combination with a simple server side detection (old pronto idea), one can reduce the traffic dramaticly. How many % you will save depends on how you integrate the server side detection and the ratio of content to non content in your project. This can be done without any inclusion of a new parameter in ajaxify, integrating one may make things easier, but has no effect on the possible traffic savings. The optional "pulse" integration will only affect the server side detection, whereas the amount of traffic savings depends on how smartly you implement the result of that detection to prevent unnecessary output.

It was my aim to share this information wth you and other interested individuals, if you will integrate anything called "pulse" does not matter to me personally, I have found a working solution for my needs at this point. I have enjoyed the discussion very much and woud like to explicitly thank you for thinking along and offering ideas. Before we get lost in global discussions, lets just leave it at this for now - I believe people familiar with custom coding, high traffic servers and SEO/page speed interests will understand the benefits of my approach, while people with a wordpress plug-n-play coding background will propably percieve this as overly complicated & nerdy. I personally don't mind the complications if I can gain 18 pagespeed score points while reducing the traffic load on my server, but maybe thats just me. There are many, many things mentioned in this discussion which you could turn into improvements of ajaxify at some point, since most of them are not relevant to my use case, I'll leave that up to you.

I'm too much of a critic of the uncontrollable behaviour of kit-built websites to approve of things like delta loading and a memory feature that seems to override the headers cache information - these are good features to set-and-forget on a static page where traffic is not of relevance, but that's simply not my ballgame. I'm affraid this discussion would always come down to this very basic difference in our points of view, which both stand to be equally right, each in their field, so lets not go there.

Bottom line - you have created something very cool which can be usefull in more situations than you might have thought :) Thanks for that!

arvgta commented 9 years ago

Thanks to you, too!

Apart from what we've established together already, one more thing came to my mind in terms of performance:

Once more, thanks for your very thorough research and incredible feedback! Do you have a URL that you would like to support, as a small token of thanks?

All the best!

arvgta commented 9 years ago

Double GET eliminated in initial load! (when deltasare enabled)

The other issue with the multiple GETs has also been tackled:

wikiloops commented 9 years ago

+1 for the delta optimizing :)

I am by now using a compressed & customized version including the pulse concept, a custom fade, the adsense concept and the additional "filter" function (as introduced in #33) on http://www.wikiloops.com

jQuery("#content").ajaxify({ fade:false,
                                          style:false,
                                          forms:"form:not(.no-ajaxy)",
                                          prefetch:false,
                                          memoryoff:true, 
                                          cb:****,
                                          inline:false,
                                          delta: false })

I'll be following your work for sure :) Two more ideas for the long run (and without opening extra issues about them):

I did mention you on the credits page of wikiloops, too, btw. Adsense experiment still on the go, I'll report for sure.

arvgta commented 9 years ago

Hey :)

Glad you like the enhancement! With the above Ajaxify call, you're using deltas right?

I don't know too much about headers, unfortunately. But your proposal is to cache them as well, right?

Prior to talking about the transitions, I would cherish, if we can handle the modifications to the existing logic, first.

(Yes, I'm aware of that "funny" of the pop and squeeze effects - these effects might need a complete overhaul)

wikiloops commented 9 years ago

There are a whole set of headers all concerned with telling a browser the desired caching effect, namedly the cache, etag, expires/max-age, pragma headers. If the cache directive tells the browser not to cache a page, or only for a given amount of time before refreshing it, it would be nice if "memory" would follow that directive.

So, the request is rather "follow caching directives" than "cache the directives to memory".

Must add: This is not nerd stuff here, this is fundamental - if no caching headers are present, files will never be cached, and most of the benefits of serving Images, CSS and JS as external code become obsolete the moment the browser does not cache them, and has to transfer them over and over again. If you have a static page that will not change at all, you may even let the html document be cached for some time. When configured correctly, your visitors can browse your site without a single request to your server and at lightspeed once they have cached all the pages. You could even turn the server off and those who cached your pages wont notice. This is all ancient practise which has been around for long, check it out sometime :)

Sorry about the misleading ajaxify call, deltas is removed from my project, so I didnt need to disable it, I'll update my post!

arvgta commented 9 years ago

Aha - you've minimised the code to run without deltas, probably stripped out the corresponding logic.

Current Ajax call:

        function _lAjax(hin, p, post, pre) { //execute Ajax load
            var xhr = $.ajax({
                url: hin,
                type: post ? "POST" : "GET",
                data: post ? post.data : null,
                success: function (h) {
                    if (!h || !_isHtml(xhr)) {
                        if (!pre) location.href = hin;
                    }
                    $.cache($(_parseHTML(h)));
                    $.pages([hin, $.cache()]);
                    if(p) p();
                },
                error: function(jqXHR, status, error) {
                    // Try to parse response text
                    try { 
                        $.log('Response text : ' + jqXHR.responseText);
                        $.cache($(_parseHTML(jqXHR.responseText)));
                        $.pages([hin, $.cache()]); 
                        if(p) p(error);
                    } catch (e) {}
                }
            });
        }

        function _isHtml(x) { //restrict interesting MIME types - only HTML / XML
            var d;
            return (d = x.getResponseHeader("Content-Type")), d && (d.iO("text/html") || d.iO("text/xml"));

The jQuery

One can supposedly fetch the cache header information with:

(mind you, there's not exactly a hell of a lot information on the web on the topic)

Algorithm

Would you be able to outline the algorithm in pseudo-code again? I'm not quite sure what it's supposed to look like? (if the response header is fetch by _lAjax(), where does the check happen etc. ?)

I believe you, that this is a rather fundamental issue - let's get it working! :)

Cheers!

wikiloops commented 9 years ago

Hey. Caching configuration by header is a quite complex thing. By exchanging headers, browser and server can negotiate one out of four possible variants:

Confusingly, the first three cases can be communicated in a variety of settings, which may be combined in several ways to achieve the desired behaviour. This is not simple but requires some knowledge about which headers do what in which hierachichal order, some headers even have identical meanings but use diffrent syntax to describe the same aspect (p.e. the max-age and expires-headers, which basicly do the same).

Because this is not simple, I'd respect it even more: webmasters who take the effort to configure their caching directives have reason to do so, so ajaxify has reason to respect these settings!

Now before going into details, and looking at the three possible variants of caching directives, one needs to ask: "In which case is it safe to use the memory extra-cache?" If the directive says "Dont cache", memory shouldn't be used. If the directive says "Cache for 60minutes" - how will you respect that? The current Memory setup propably does not feature some "expires" clock, so we are going to be in trouble here. If the directive says "get back to me and get my OK before serving from cache", memory shouldn't be used either, because it will prevent this planned validation...

From my point of view, this is what is quite problematic about memory, A conservative approach would come down to: If there are caching directives, if cache control is "public" and if max-age or expires headers indiciate the page will be fresh for over 24hours to come and no validation is intended, it is safe to store the page to memory. The 24hours part is an assumption based on the idea that propably only few people will keep a browser open longer than 24hours, and since memory will loose all cached pages on browser close, we are on the safe side here. Now, in the end, these conditions will be met by a very, very small amount of pages on the web, so setting a directive like this will disable memory on most pages with caching directives set.

As long as you dont want to do something really comlplex to adopt the exact caching behaviour to memory, I'd take a big step back and just check for the presence of the "cache-control" header. If it is present, disable memory, if its not, well, do as you please :)

As for the pseudo logic: The check for headers must happen in the success-function (otherwise, there will be no headers received). If the headers are present and indicating caching is too complex to be handled by memory, I'd just prevent the $.cache($(_parseHTML(h))); $.pages([hin, $.cache()]); - part. the page will be handled as expected, just no storing to memory taking place.

Sorry to say it, but the longer I think about it, your setup of memory is really only helpfull for sites with poorly configured headers - if there are no cachng directives, a webmaster may feel he is profiting from memory. On a site with delicatly chosen caching settings, the browser cache does the job of memory without any need for javascript on top of that. I dont believe there is an "out-of-the-box" solution for this, and if there was one, it should be dealing with headers, not working around "on top".

arvgta commented 9 years ago

Thanks for the detailled reply!

Let's not see things so pessimistic. What is available on the client side by means of the plugin, should be in addition to what is possible from the server side. Let's not forget, that there is the client-side possibility to specify several pages that should not be cached. After all, some users may not be that acquainted with setting headers etc. Now, what is missing is the "respecting" of the server-side headers - alright.

I'll start with implementing the simplest solution, that you mentioned (the last variant).
I've inserted the following code:

                    if(!xhr.getResponseHeader("Cache-control")) {
                        $.cache($(_parseHTML(h)));
                        $.pages([hin, $.cache()]);
                    }

enable in this test file only at the moment

Please test...

If you find this useful, I'd be eager to specify a more detailled condition.

Thanks and regards.

wikiloops commented 9 years ago

well, this is not about pessimism - as you said it: it should be in addition, and not interfering. The beauty of cache configuration by header is that it is very, very flexible - unlike the global "never cache page A"-directive you are offering. You can do "use cached page as long as there is no new comment" moves by header configuration, to give one use case, or differentiate caching behaviour by user or device.

Your fix is fine, but it will disable memory on many decently coded pages - just browse some well known pages and have a look at the headers they sent by dev console and see for yourself. if you feel your target audience are people who don't go for search engine ranking, speed and saving server cost but need easy tools for their hobby page, your approach is still cool with me. I would definetly add a "works as fallback if no caching directive is sent" to the "memory" enabler, so people dont get confused why the option they might have enabled doesn't do anything for them.

Your fix should propably be applied to the preload, too, in case that stores the prefetched page to memory as well...

I am sorry if this reads like I'm dissing the stuff you developed, I am just giving you the reasons why I had to disable all those well developed features :/ I just believe you would possibly benefit from not confusing the more advanced users, so I'd try to make sure the default settings of ajaxify dont interfere with anything an "expert" would have solved properly - even if that means a non expert will have to set memory & preload to true if really desired. Its more of a strategic & user experience question concerning your plugin than pessimism :)

arvgta commented 9 years ago

Hi Wikiloops!

Sorry for the delay, there's quite a lot going on here...

Before we go on, could you please test this file against your server, where Cache-control is set somethimes?

I'm asking because when I first implemented it, I forgot about the "!" in the following snippet:

                    if(!xhr.getResponseHeader("Cache-control")) {
                        $.cache($(_parseHTML(h)));
                        $.pages([hin, $.cache()]);
                    }

and the result was chaotic!

So, I reckon, that latest change will not do the job...

You're right - steering memory from the server-side is very elegant, flexible and powerful. The first thing we need is the right hook in the present algorithm...

Also, #32 is ready for testing :)

Thanks in advance and Regards

arvgta commented 9 years ago

@wikiloops : Thanks for all your contributions! Could you please indicate, whether you would like to continue our cooperation? If not, I will rollback the above change as it doesn't work.

Thanks and Regards

arvgta commented 9 years ago

Last bit concerning Cache-control rolled back. Now, all files @4nf.org and GitHub ought to be roughly the same functionality.

If someone has a concept how to handle Cache-control properly, feel free to open a new, dedicated issue...

Thanks!