[RS] I want to crawl reddit and complete a draft design for review

[johnklee]

The first site I want to crawl is "https://www.reddit.com/". Below is the CUJs to consider in designing our crawler:

• I want to store the crawled result in DB (Support postgresql in default) for post process
• I want to configure the crawler to decide what content I am interested in and drop the uninterested page(s)
• I want to configure the crawler to support throttling control. (e.g. Processed N page per minutes at most)
• I want to support Robots exclusion standard.
• I want to configure specific URL patterns and only crawl the page(s) match given URL patterns to avoid unnecessary traffic/requests
• I want crawler to support customized/basic text extraction (html -> text).
• ...

[johnklee]

Information gathering/collection for design reference:

• 5 Popular Python Libraries to Perform Web Scraping