search

iriscouch / follow

Very stable, very reliable, NodeJS CouchDB _changes follower
Apache License 2.0
393 stars 82 forks source link

`follow` does not follow `skimdb.npmjs.com` to the end #74

Closed nknapp closed 8 years ago

nknapp commented 8 years ago

I am currently writing a program to import the npm-registry into another database. I have the program that follow stops at change-number 375253 and does not receive any more data, but skimdb has over 500000 changes right now. The issue can be reproduced with the following code snippet:

var follow = require('follow');

follow({db: "http://skimdb.npmjs.com/registry", since: 375250, include_docs: true}, function (error, change) {
    console.log("received seq " + change.seq + " from update stream");
});

I'm getting the output 

received seq 375253 from update stream

followed by nothing but a waiting console (for days). However, if I go to https://skimdb.npmjs.com/registry/_changes?feed=continuous&since=375250&include_docs=true I get many more updates.

It would by nice if you could have a look into it.

nknapp commented 8 years ago

Mhm the document coming along with change 375255 seems to be rather big (4.9 MB). Maybe that has something to do with it.

nknapp commented 8 years ago

I'm closing this issue, because the example above seems not to be enough to reproduce the problem. After waiting a while, stream continues. It just has a longer pause, which might be caused by the large document. My actual program is a bit more complicated and the problem could not be solved by just waiting, but I will have to try to create a new minimal example to reproduce it.

nknapp commented 8 years ago

This seems to be an issue with node-version 0.12.7. The current stable (4.2.3) produces the behavior described in my last comment: About 20 seconds of being very busy (100% CPU) and then continuing.