search

tether / roach

A very adaptable web crawler framework. Impossible to kill.
Other
7 stars 1 forks source link

We should have a CLI for scaffolding a crawler #7

Closed ekryski closed 10 years ago

ekryski commented 10 years ago

Crawlers will typically be placed in a crawlers directory. The structure should look something like this:

  crawlers
    -> example_crawler
      -> roach.json
      -> index.js
      -> steps
        -> custom-step1.js
        -> custom-step2.js