search

fent / node-feedsub

Subscribes to RSS/Atom/JSON feeds and notifies on new items.
MIT License
196 stars 19 forks source link

Support paged feeds #20

Closed StarpTech closed 7 years ago

StarpTech commented 7 years ago

Hi, it is common that a feed can support paging by defining a next link. It would be very efficient if we could take advantage of that.

https://tools.ietf.org/html/rfc5005#section-3

fent commented 7 years ago

This module is meant to be used as subscriber-like, with interest mostly on new items as they get published.

StarpTech commented 7 years ago

yes I understand but it would be very efficient to process a small page of feeds instead to iterate over thousand of items even when they are already outdated. This is the job of the subscriber because its part of the protocol.

fent commented 7 years ago

In case of a paged feed, it will only look at the first page. And it will already not process further items and abort the download if it encounters one it has before.