search

taganaka / polipus

Polipus: distributed and scalable web-crawler framework
MIT License
92 stars 32 forks source link

[WIP] Plugins architecture proposal #46

Open taganaka opened 10 years ago

taganaka commented 10 years ago

cc @tmaier

coveralls commented 10 years ago

Coverage Status

Coverage decreased (-0.62%) when pulling 02ee4841c3280f8316ee805d33da29ce0f22fbf1 on plugins into 0e4b19adbce8d408994ce83b43ea6fe148821a55 on master.

taganaka commented 10 years ago

API Proposal:

Polipus.crawler(name, starting_urls, options) do |crawler|
  crawler.register_plugin(Class::Name, plugin_options)
end
tmaier commented 10 years ago
  1. Shall a plugin be able to be registered more than once?
  2. Shall it be possible to unregister a plugin again? This way, we could register some default plugins. If someone does not like them, they can unregister them by themself. This would also be necessary to stay backwards compatible: We automatically register the plugins resulting from the old options. And give a deprecation warning.