adding initial roadmap document

[b5]

Feel free to add to this! Comments Welcome!

closes #6.

[Mr0grog]

Milestone proposals:

• Single application that can crawl a full web site or a simple list of URLs and store the results on disk
• Swappable workers (one of which uses Headless Chrome)
• Many pluggable result handlers (not just on-disk storage)
• Distributed operation with workers on separate machines

Should we just make milestones or should we also add those to this doc?

[Mr0grog]

Also:

• Deploy it on a server somewhere to run regularly
• Long-running process that can be scheduled to do different domains, page lists, etc. at different schedules
• (HTTP?) API for setting up schedules/domains/page lists and querying results/stored data
• External push notifications (probably web hooks)
• Web UI for configuring schedules/domains/page lists

[b5]

My only concern with adding proposed milestones to the docs is that we'll forget to update them. But I think we should add them. Even if they fall out of date, I agree with all those milestones, and think we should have them in the roadmap to show our long-term-but-concrete intent for the project. Let's add 'em!

[codecov-io]

Codecov Report

Merging #10 into master will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #10   +/-   ##
=======================================
  Coverage   60.67%   60.67%           
=======================================
  Files           8        8           
  Lines         417      417           
=======================================
  Hits          253      253           
  Misses        128      128           
  Partials       36       36

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 89fe497...733adef. Read the comment docs.

[Mr0grog]

Done and done.