Error listing; Number of URLs; Documentation; Virtualenv support

Webdevdata / fetcher

Tool to download website data.

The Unlicense

9 stars 4 forks source link

Error listing; Number of URLs; Documentation; Virtualenv support #7

Closed matatk closed 9 years ago

matatk commented 9 years ago

I've been running this script for @stevefaulkner and, as part of this, I made the following changes and thought you might like to consider them for upstream.

Keep track of site access errors in a file in the output directory
Only grab the top 100,000 URLs by default
Update and re-wrap documentation
PEP8 adherence
Support virtualenv for managing the Pyth

yoavweiss commented 9 years ago

Normally, I'd prefer smaller PRs that are easier to review.

Nevertheless, thanks for all your work. LGTM :)

matatk commented 9 years ago

You're welcome -- and, yeah, it's a good idea to split things up a bit more; I'll do so with any future PRs.