Possible problems with Meteor.isReadyForSpiderable

[naho2081]

I have a concern regarding Meteor.isReadyForSpiderable variable.

It seems to me that phantom_script.js that is placed by package settings under Meteor Assets (full path "#{Meteor.rootPath}/assets/packages/jazeee_spiderable-longer-timeout/lib/phantom_script.js" doesn't even have access to such variables as Meteor, Package. That's why it can't "read" Meteor.isReadyForSpiderable variable.

Reasons why I think so:

1. In our project routes that even don't have Meteor.isReadyForSpiderable = true setting at all are "spidered" correctly.

2. Pages that are normally loaded in less than 300ms are "spidered" with ?_escapedfragment= longer than 4 seconds.

3. If in if(totalIterations > 200) setting in the phantom_script.js change totalIterations value to 50 or 100, pages are spidered much more quickly, but in some cases spiderable returns page in "loading" state that tells us that not all subscriptions are ready. Again, even if corresponding route has Meteor.isReadyForSpiderable = true setting after @ready.

See attached screenshots with load time for the page with ?_escapedfragment= and without it.

Who has seen similar behaviour or am I missing something?

Waiting for server response with ?_escapedfragment= Total time WITH ?_escapedfragment= Total time WITHOUT ?_escapedfragment=

[dr-dimitru]

Hey @naho2081 ,

What service are you using? I would like to test our sites too. Anyways, anything below 10sec response is okay for "prerendered" pages. Is a way PhantomJS works, it's slow. What about second request?

[naho2081]

Hey @dr-dimitru ,

that is free Pingdom website speed tool available here. And it supports rendering of Meteor pages without ?escaped_fragment=.

In my example second screenshot shows total loading time for same page, but without ?escaped_fragment=.

Added here screenshot that shows how it loads. As you see, approximately after 500ms it starts downloading images. This means that till this time HTML is fully ready on the client side.

PhantomJS is not so slow that it will render same page 4 seconds. With adjusted setInterval and totalIterations in phantom_script.js we achieved page rendering time by PhantomJS that is around 300 ms instead of almost 4s.

[dr-dimitru]

Hello @naho2081 ,

I got interesting results:

1. Direct link (https://veliovgroup.com) loads few seconds - I assuming Pingdom uses PhantomJS or similar on their end
2. With escaped query (https://veliovgroup.com/?_escaped_fragment_=) - This page is in cache of course but result is great - ~580ms
3. With escaped query to non-existent page (to make sure it's not cached) (https://veliovgroup.com/not/exists/?_escaped_fragment_=) - always less than 2s, which is pretty good result.

We are using global IS_RENDERED variable to catch if page is ready. It adds smallest overhead, as requires to add into logic a bit deeper, than router level.

Also all CSS is passed through in this tests. And JS is removed form HTML.

[naho2081]

Hey @dr-dimitru ,

could you please clear Spiderable cache (empty Meteor collection SpiderableCacheCollection) and test again https://veliovgroup.com/?_escaped_fragment_= ? How much time it will show now?

[dr-dimitru]

Sure, see attached image. Needs to be said - it has delay for animation, I mean spiderable returns result only after animation is finished, and it's around extra ~1.5sec. So, if total result is 3.48sec the rendering is ~1.98sec.

[naho2081]

I have optimized Spiderable settings and now rendering occurs much faster and is under 2 seconds almost for all pages, even with large number of subscriptions.

As I have assumed above, PhantomJS doesn't have access to Meteor variables and can't read Meteor.isReadyForSpiderable variable. All what it has is HTML code produced by server.

In the phantom_script.js I have replaced isReady function with following one:

var isReady = function (){
    html = page.content
    if(html.indexOf("loading__completed") !== -1){
      //console.log("Loading completed");
      return true;
    }
  }

This function looks for _loading_completed text in HTML. This text I insert into the page with separate template when all Subscriptions are ready and when all other templates are being rendered:

div(class='hidden')
    <!-- loading__completed -->

So, as soon as PhantomJS sees in HTML returned from server _loading_completed it stops and returns page.

[dr-dimitru]

@naho2081 good solution, similar to how we use IS_RENDERED variable