rgarner / cma-tna-crawlers

Scraping old cases from TNA for CMA, no TLAs.
0 stars 3 forks source link

CC crawler not saving subpages #11

Closed rgarner closed 9 years ago

rgarner commented 9 years ago

Missing things like evidence/news in the larger cases, e.g. http://webarchive.nationalarchives.gov.uk/20140402141250/http://www.competition-commission.org.uk/our-work/directory-of-all-inquiries/aeg-wembley-arena