danmactough / node-feedparser

Robust RSS, Atom, and RDF feed parsing in Node.js
Other
1.97k stars 192 forks source link

With interval, a request piped to feedparser only works first time #206

Closed Eightquake closed 7 years ago

Eightquake commented 7 years ago

I have a request to a feed that I pipe to feedparser, the request is sent every 6 minutes. The feed link is in Swedish, but it is a valid RSS feed.

/* Inside a function called by an interval */
request
    .get('http://www.sweclockers.com/feeds/forum/trad/999559')
    .on('error', function(err) {
      /* handles error */
    })
    .on('response', function(response) {
      /* uses maybeDecompress function from example */
    })
    .pipe(feed);

/* Outside of any function in executed .js-file */
feed.on('readable', function () {
  /* Uses the code from the example in README.md */
}

This works without a problem the first time: the request is sent, response is received and feed parses it and does what it should do. The next time this code is run, the request is sent and the response is received and then nothing. No readable event is ever fired after this, even if new items has been added to the RSS feed.

I have a function that logs these events: request errors and responses, feedparser errors and readable, and interval calling the request-function. This is my log for the events:

2017-03-09 01:18:57 | Interval | Fetching the feed
2017-03-09 01:18:57 | Request | Response
2017-03-09 01:18:57 | Feed | Readable
2017-03-09 01:18:57 | Feed | Readable
[...]
2017-03-09 01:18:57 | Feed | Readable
2017-03-09 01:24:57 | Interval | Fetching the feed
2017-03-09 01:24:58 | Request | Response
2017-03-09 01:30:58 | Interval | Fetching fynd
2017-03-09 01:30:58 | Request | Response
2017-03-09 01:36:58 | Interval | Fetching fynd
2017-03-09 01:36:58 | Request | Response
2017-03-09 01:42:58 | Interval | Fetching fynd
2017-03-09 01:42:58 | Request | Response
2017-03-09 01:48:58 | Interval | Fetching fynd
2017-03-09 01:48:58 | Request | Response

You should note above that the first time, the request get's a response, and feedparser emits many readable events, which I have truncated with [...]. But after that, the request get's a response but feedparser does nothing. I am running Node v6.9.2 and Feedparser is version 2.1.0.

Do I need to drain/clear the stream before I fetch the feed again or something like that? It works as intended the first time, but not after that. I can restart the script and run it multiple times within 6 minutes, so the problem isn't the server rejecting the connection. I couldn't find any issue similar to this, but if you know of one please show me.

danmactough commented 7 years ago

Yeah, the feedparser stream will be closed and ended when it's finished. You may be able to reuse the same stream for multiple requests somehow, but I would just create a new feedparser instance for each request you want to handle.

Eightquake commented 7 years ago

Cheers, I did what you said - created a new Feedparser instance and that worked.