v1.1.9

[dr-dimitru]

• Add Spiderable.query {Boolean|String} - property
  • Spiderable.query {Boolean|String} - additional get query appended to http request.
  • This option may help to build different client's logic for requests from phantomjs and normal users
• Docs update

[dr-dimitru]

Added partly fix for issue described in #4 We're still expecting Error serving static file Error: Requested Range Not Satisfiable and sometimes caching was enabled after some bytes had been written, but meteor and node is not fails on fiber issue, so at least application is up and running without nodejs restart.

[dr-dimitru]

v1.2.0

• Add caching via mongoldb collection
• TTL for cached results via Spiderable.cacheTTL
• Get rid of fs
• Cleanup code

Tested on our production stage, caching gives headshot speed up to this package

[dr-dimitru]

@jazeee please let me know when you will be ready to move on coffeescript

[dr-dimitru]

Anyways, thank you for accepting PR, waiting to update it via Atmosphere. And for the moment to move on CoffeeScript.

[jazeee]

I have published to Atmosphere, and things seem to work well. Thanks for the extra work. Remember I made some API name changes to be clear.

I have converted to Coffeescript and tested as well. (I broke a path, since the phantomJS script is now in lib)

(version is now 1.2.2)

[dr-dimitru]

Okay, thank you. I will update my fork and test on my end tonight.

Also I would like to enhance rep's docs, preferably from the scratch, so let me know if you have any ideas and need help on it.

[jazeee]

By the way, when I converted to coffeescript, I also simplified one area of code, which had a nested if statement. See around https://github.com/jazeee/jazeee-meteor-spiderable/blob/master/lib/server.coffee#L93

I don't use the ignoredRoutes option, but I believe it should still work as it did before.

[dr-dimitru]

@jazeee Not okay, getting this, where supposed to be 404:

[CRITICAL] QNetworkReplyImpl: backend error: caching was enabled after some bytes had been written

Can't fix, any ideas?

[jazeee]

No ideas. I'll have to test tomorrow. Try with a minimal meteor project to see if it is spiderable or code. On Jul 22, 2015 6:27 PM, "dr.dimitru" notifications@github.com wrote:

@jazeee https://github.com/jazeee Not okay, getting this, where supposed to be 404:

[CRITICAL] QNetworkReplyImpl: backend error: caching was enabled after some bytes had been written

Can't fix, any ideas?

— Reply to this email directly or view it on GitHub https://github.com/jazeee/jazeee-meteor-spiderable/pull/7#issuecomment-123905230 .

[dr-dimitru]

@jazeee error on my end. Anyways I'm thinking about to retranslate response code via our package - this is what people waiting for. Yesterday I saw on many forums what almost no one solve the issue serving 404 to the bots

[jazeee]

I don't see the error you described, but rather I see that PhantomJS times out.

Based on my testing, if the webapp returns with 404, it will not set Meteor.isReadyForSpiderable to true.

This means that phantomJS is waiting for the code to finish, but never gets past: https://github.com/jazeee/jazeee-meteor-spiderable/blob/master/lib/phantom_script.js#L19

After 30 seconds, the server will decide that PhantomJS will not respond and kills it.

I believe we need to add some code to phantom_script.js that checks for 404 response code (or 4xx response codes) and if detected, respond with that.

I'll look at some ideas on this.

[jazeee]

I created #9 to document the 404 issue. I have some code that should help address it, albeit not in an ideal manner...