search

djett41 / node-feedjett

A fast customizable RSS, ATOM, and RDF feed parser and aggregator
MIT License
5 stars 2 forks source link

readme

Build Status

NPM

FeedJett - A Fast Robust RSS, Atom, and RDF feed parser in Node.js - Like a Jett ;-)

This module is inspired by Dan Mactough's node-feedparser module which parses RSS, Atom, and RDF feeds using Isaac Schlueter's sax parser.

FeedJett is built on much of the same model for sax parsing as feedparser and shares alot of the same boiler plate code to parse feeds, though it does contain many enhancements and features as listed below

Enhancements / Features of FeedJett

Installation

npm install feedjett

Usage

The easiest way to use feedjett is to just give it a readable stream. It will then return a readable object stream.


var FeedJett = require('feedjett'),
    request = require('request');

var req = request('http://somefeedurl.xml'),
    feedjett = FeedJett.createInstance([options]);

req.on('error', function (error) {
  // handle any request errors
});
req.on('response', function (res) {
  var stream = this;

  if (res.statusCode != 200) return this.emit('error', new Error('Bad status code'));

  stream.pipe(feedjett);
});

feedjett.on('error', function(error) {
  // always handle errors
}).on('meta', function (meta) {
  // emits the meta event if parseMeta is true (defaults to true)
  console.log(meta);
}).on('readable', function() {
  // This is where the action is!
  var stream = this, item;

  while (item = stream.read()) {
  //item will also have a meta object attached to it if addMeta is true (defaults to true)
    console.log(item);
  }
});

I strongly encourage you to take a look at the demo test case for a more thorough working example.

How FeedJett works

FeedJett parses exactly what you want and how you want. By default FeedJett defines Arrays of properties for meta that dictate what the result meta or item should look like for all feeds (assuming normalize is set to true). These Arrays of properties are defined in an object called normalizedProps (one for meta, one for item). Each property in the Array will map to a parsing function which is responsible for parsing and normalizing node values into the defined property. This way, we aren't looping through each and ever property in the node, but instead are just parsing what needs to be parsed and nothing more. Default normalized props and parsing functions can be tailored and customized as described in the next section.

Options

When creating a new instance of FeedJett, you can pass in an options object with one or more of the following object properties

NOTE: some of the following options may be similar to FeedParser but syntactically different (camelCased in most cases)

options continued.. (customizations)

Add the following to the options object to further customize the parsing framework for FeedJett. These options and customizations assume that the normalize option is not set to false.

Normalized Properties Customizations

NOTE: You can only define either blackList OR whiteList on options. If both are defined, whiteList will take precedence


var options = {
  blackList: ['description', 'updatedDate']
};

//OR!! Don't define both!!!

var options = {
  whiteList: ['title', 'description', 'link', 'pubDate']
};
var feedjett = FeedJett.createInstance(options);  //... other feedjett logic as shown in earlier section

Parsing Function Customizations

The following object (defined in feedjett.js) maps normalized props to parsing functions defined in the parser.js file. You will also find detailed parsing information and logic for each parsing function.


var propertyParserMap = {
  title: 'parseTitle',
  description: 'parseDescription',
  link: 'parseLink',
  updatedDate: 'parseUpdatedDate',
  pubDate: 'parsePubDate',
  categories: 'parseCategories',
  author: 'parseAuthor',
  image: 'parseImage',
  xmlUrl: 'parseXmlUrl',
  origLink: 'parseOrigLink',
  language: 'parseLanguage',
  icon: 'parseIcon',
  copyright: 'parseCopyright',
  generator: 'parseGenerator',
  cloud: 'parseCloud',
  comments: 'parseComments',
  commentRss: 'parseCommentRss',
  updateInfo: 'parseUpdateInfo',
  enclosures: 'parseEnclosures',
  guid: 'parseGuid',
  source: 'parseSource',
  content: 'parseContent'
};

If the parsing logic for a specific parsing function doesn't suit your needs, then you can easily override the function with your own. For example, if you would like to parse the normalized description property, you can add the following function to the options object, (must match the parsing function name) which will then be used to parse and normalize the description property.

NOTE: each parsing function can define the following parameters be passed in the following node


var options = {
  parseDescription: function (node, nodeType, feedType) {
    //add your custom parsing logic here

    return 'custom parsed string value!';
  }
};
var feedjett = FeedJett.createInstance(options);  //... other feedjett logic as shown in earlier section

Custom Property names via FeedJett.overridePropNames

FeedParser provides a static function for overriding the names of the default normalized properties. All you have to do is pass in a mapping object and the feed item/meta will have your custom property name instead of the default.
The mapping object keys should be the original property name, and the value should be your custom property name

Example..


FeedJett.overridePropNames({
  description : 'summary',
  link : 'url',
  pubDate : 'pub_date',
  guid : 'id'
});

var feedjett = FeedJett.createInstance();  //... other feedjett logic as shown in earlier section

Now when you listen to readable for items or the meta event for meta, the result object will include myProperty which would have a value that equals the return value of parseMyProperty. Behind the scenes, addCustomParser will add the property to the normalizedProps meta or item Arrays, then add the myProperty -> parseMyProperty mapping to the propertyParserMap object, then dynamically invoke the custom function and set the property on the result object.

Custom Properties and Parsers via FeedJett.addCustomParser

FeedJett provides a static function for adding custom properties not avialable by FeedJett as well as providing a custom parser function to normalize the property. To achieve this you can call the FeedJett.addCustomParser, passing in the custom property name, nodeType for the custom property, parser function name, and parser function.

For example, if you wanted to add a custom normalized property to both meta and item objects called myProperty, you would write the following code.


FeedJett.addCustomParser('myProperty', ['item', 'meta'], 'parseMyProperty', function (node, nodeType, feedType) {

  //custom logic for parsing and normalizing myProperty.  For example if you wanted the text value of atom:id to be
  //the value of myProperty, you would do the following.. ('#' are for text values, @ for attribute objects)
  return node['atom:id'] && node['atom:id']['#'];
});

var feedjett = FeedJett.createInstance();  //... other feedjett logic as shown in earlier section

Now when you listen to readable for items or the meta event for meta, the result object will include myProperty which would have a value that equals the return value of parseMyProperty. Behind the scenes, addCustomParser will add the property to the normalizedProps meta or item Arrays, then add the myProperty -> parseMyProperty mapping to the propertyParserMap object, then dynamically invoke the custom function and set the property on the result object.

Custom Parsing Hooks

You can also include functions that can hook into the parsing framework. The following functions can be defined on the options object to achieve this

The following example illustrates how you can hook into the parsing framework AFTER an item or meta is parsed and define any additional logic. The raw node (item or meta) and feedType (rss, atom or rdf) will be available as parameters. No return value is needed as you can add properties on the meta or item object.


var options = {
  afterParseItem: function (item, feedType) {
    //custom logic.. set link to guid if the parsing functions couldn't normalize link
    if (!item.link) {
      item.link = item.guid;
    }
  },
  afterParseMeta: function (meta, feedType) {
    //custom logic.. set updatedDate to pubDate if the parsing functions couldn't normalize updatedDate
    if (!meta.updatedDate) {
      meta.updatedDate = meta.pubDate;
    }
  }
};
var feedjett = FeedJett.createInstance(options);  //... other feedjett logic as shown in earlier section

The following example illustrates how you can override the default behavior to determine whether or not an item or meta is valid. The raw node (item or meta) and feedType (rss, atom or rdf) will be available as parameters. A boolean value true or false must be returned


var options = {
  isMetaValid: function (meta, feedType) {
    //custom logic.. mark meta valid and emit meta only if feedType is rss..
    return feedType === 'rss';
  },
  isItemValid: function (item, feedType) {
    //custom logic.. mark an item as valid only if it contains a pubDate
    return !!item.pubDate;
  }
};
var feedjett = FeedJett.createInstance(options);  //... other feedjett logic as shown in earlier section

Utils functions

FeedJett comes equipped with a variety of utility functions for parsing feeds. To reuse a utility function in your custom parser(s), get an instance of the utils module as shown below

var utils = FeedJett.utils;

var options = {
  parseAuthor: function (node, nodeType, feedType) {
    return utils.getFirstFoundPropValue(node, ['author', 'dc:creator', 'itunes:author', 'dc:publisher']);
  }
};
var feedjett = FeedJett.createInstance(options);  //... other feedjett logic as shown in earlier section

Examples

See the demo directory.

API

Transform Stream

Like FeedParser, FeedJett is a transform stream operating in "object mode": XML in -> Javascript objects out. Each readable chunk is an object representing an item (article) in the feed.

Events Emitted

FeedJett Normalized properties

FeedJett parses each feed into a meta (emitted on the meta event) portion and one or more item (emitted on the data event or readable after the readable is emitted).

Regardless of the format of the feed, the meta and each item contain a uniform set of generic properties patterned after (although not identical to) the RSS 2.0 format, and optionally all of the properties originally contained in the feed. So, for example, an Atom feed may have a meta.description property, but it could also have a meta['atom:subtitle'] property.

The purpose of the generic properties is to provide the user a uniform interface for accessing a feed's information without needing to know the feed's format (i.e., RSS versus Atom) or having to worry about handling the differences between the formats. However, the original information is configurable in case you need it. In addition, FeedJett supports some popular namespace extensions (or portions of them), such as portions of the itunes, media, feedburner and pheedo extensions. So, for example, if a feed article contains either an itunes:image or media:thumbnail, the url for that image will be contained in the article's image property.

Unlike FeedParser, FeedJett does NOT pre-initialize generic properties and keeps result objects as thin as possible. Additionally, result object properties are camelCased. In contrast, the original raw node will contain lowercase property names (which is how sax parses the nodes). So for example if you are defining a custom function to parse the pubDate normalized prop, you look for node.pubdate in the Raw node, but the normalized object will have the property item.pubDate.

The title and description properties of meta and the title property of each item have any HTML stripped if you let FeedJett normalize the output. If you really need the HTML in those elements, there are always the originals by passing in parseRawNode: true in options and accessing the original property in the meta.raw object. e.g., meta['atom:subtitle']['#'].

List of normalized meta properties (similar to RSS channel, or ATOM feed)

NOTE:

  1. meta object's will also always include the following properties

List of normalized item properties (similar to RSS item, or ATOM entry)

NOTE:

  1. If parseMeta and addMeta are both true, item will include a meta property that includes meta properties.
  2. If parseRawNode is true, meta will include a raw property that contains all raw nodes from the feed
  3. FeedJett content, and updatedDate are the same as FeedParser summary, and date. (Although I believe more accurate ;-)). FeedJett adds commentRss, but drops support for permalink.

Issues