Testing

[samliu]

• Tested all added middlewares (we only added spidermiddlewares) to make sure only the correct requests get filtered. -- Depth 2 requests are not filtered (linked from base 1, a.k.a base links) -- Depth 3 requests are filtered if they're html -- Depth 3 requests are not filtered if they're css/js -- Offsite requests are filtered
• Tested the spider's crawling mechanism to ensure -- The correct items are emitted -- Different ua strings are used correctly -- Reading from a text file works
• Tested model constraints

[carljm]

Just merged this - looks great! You're now at about 87% test coverage. If you want to take a look at the coverage yourself, here's how:

pip install coverage
coverage run --branch runtests.py     # runs the tests under coverage monitorying, with code-branch tracking
coverage html --include='spade/*' --omit='spade/tests/*'    # we want to see coverage for spade/ except for spade/tests
firefox htmlcov/index.html    # "coverage html" outputs by default into "htmlcov/"

If you feel like it, you could add a script to do some of this into the repo, and add htmlcov to .gitignore.