Closed spulec closed 10 years ago
I think res.ok
is probably a synonym for 2XX
. So maybe if res.status_code >=200 and res.status_code < 400
, to catch non-error states?
This is great, feel free to merge after making those changes, and testing the scraper in at least one condition where the url
field is missing.
(I'm going to be busy merging your IG scrapers for a bit.)
It appears that res.ok
basically means res.status_code not in range(400, 601)
. See here.
I think this means we are good to switch to it.
:+1:
This is what I've been using locally for the past few scrapers.
I don't think
check_report_url
is a particularly great name.It's also possible that we should allow other status codes (301, 302, etc). Maybe use
res.ok
instead?Closes #108