As a simple test, I replace that line with if(renderIterations < 30 ){ which is the equivalent of waiting 3 seconds before processing the response. When I do that, I get the correct 404 response. If I change it to 20, it responds with a 304 or occasionally 404 or 200.
If I do the same on a smaller Meteor project, I always see 404, so I conclude that it is dependent on project size, and probably CPU and system load.
One thing that we may be able to do is be more reliant on Meteor.isReadyForSpiderable, which should only be set once the route has completed and all subscriptions are ready.
@dr-dimitru - These are my thoughts on the issue.
In my testing, the phantom script behaves differently depending on the server load and CPU.
In particular, I have found that the following area of code seems to respond differently depending on how long one waits before responding. https://github.com/jazeee/jazeee-meteor-spiderable/blob/master/lib/phantom_script.js#L72
As a simple test, I replace that line with
if(renderIterations < 30 ){which is the equivalent of waiting 3 seconds before processing the response. When I do that, I get the correct 404 response. If I change it to 20, it responds with a 304 or occasionally 404 or 200.If I do the same on a smaller Meteor project, I always see 404, so I conclude that it is dependent on project size, and probably CPU and system load.
One thing that we may be able to do is be more reliant on
Meteor.isReadyForSpiderable, which should only be set once the route has completed and all subscriptions are ready.I will test this approach...