taganaka / polipus

Polipus: distributed and scalable web-crawler framework
MIT License
92 stars 32 forks source link

Build Status Coverage Status Code Climate RubyGems

Polipus

A distributed web crawler written in ruby, backed by Redis This project has been presented to the RubyDay2013 http://www.slideshare.net/francescolaurita/roll-your-own-web-crawler-rubyday

Features

Supported Ruby Interpreters

Survival code example

require "polipus"

Polipus.crawler("rubygems","http://rubygems.org/") do |crawler|
  # In-place page processing
  crawler.on_page_downloaded do |page|
    # A nokogiri object
    puts "Page title: '#{page.doc.css('title').text}' Page url: #{page.url}"
  end
end

Installation

$ gem install polipus

Testing

$ bundle install
$ rake

Contributing to polipus

Copyright

Copyright (c) 2013 Francesco Laurita. See LICENSE.txt for further details.