search

rgarner / cma-tna-crawlers

Scraping old cases from TNA for CMA, no TLAs.
0 stars 3 forks source link

Work out why so many cases missing from competition/cartels #24

Closed rgarner closed 9 years ago

rgarner commented 9 years ago

Output from sheets suggests we need another crawler, as our only current competition/ca98 cases come from the current cases page/completed links - propose starting here instead:

/OFTwork/competition-act-and-cartels/ca98/ /OFTwork/competition-act-and-cartels/criminal-cartels-completed/ OFTwork/competition-act-and-cartels/ca98/decisions/

Augmenting with sheets/ca98-cartels/OFT CA98 and civil cartels-Table 1.csv
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/LME' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/ABI  http://webarchive.nationalarchives.gov.uk/20140402142426/http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/abihttp://webarchive.nationalarchives.gov.uk/20140402142426/http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/abi' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/online-retail/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/business-continuity/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98-current/commercial-vehicle-civil/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/leisure-goods-2/#named1' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/local-buses/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/nationwide' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/Oakley' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/transport-sector/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/retailers-and-suppliers-ccs1' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/visa-olympic-sponsorship/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/Walkers#_note-see note 1' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/British-Airways2' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/street-furniture/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/cardiffbus' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/fuel-surcharges/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/bars' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98-current/dairy-products/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/scotland-roofing' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/flat-roof' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/wm-roofing' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/hasbro' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/schools' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/john-bruce' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/lladro' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/loan-services/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/scotland-flat-roofing' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/Mercedes-Benz' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/football-kit' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/pads' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/tobacco' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/Construction-recruitment-forum' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/bid_rigging_construction AND  http://webarchive.nationalarchives.gov.uk/20140402142426/http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/construction/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/argos2' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/arriva' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/ne-roofing' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/healthcare-products/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/recruitment-agencies/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/uop-limited' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/criminal-cartels-completed/fuel-surcharges-proceedings' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/genzyme' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/livestock' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/reckitt-benckiser' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/aberdeen-journals ' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/anaesthetists-groups' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/attheraces' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/bunker-fuels/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/companies-house' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/consignia' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/du-pont' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/elite' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/lothian' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/flybe' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/icl' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/veterinary-diagnostics/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/link' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/lucite' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/bettercare' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/belfast' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/pool' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/oil-supply' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/bskyb  ' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/general-insurance' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/harwood' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/criminal-cartels-completed/agricultural-sector/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/British-Airways' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/ba' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/dsg' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/e-books' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/fda' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/ncrcorporation' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/network-rail' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/portec' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/motor-insurance' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/proctorgamble' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/scottish-milk' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/sports-goods/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/swan' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/tm-property' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/closure/virgin-cathay/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/anl' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/ca98/decisions/napp' not in index
Augmenting with sheets/ca98-cartels/OFT criminal cartels-Table 1.csv
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/criminal-cartels-completed/automotive-sector/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/criminal-cartels-completed/commercial-vehicle-criminal/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/criminal-cartels-completed/fuel-surcharges-proceedings' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/criminal-cartels-completed/agricultural-sector/' not in index
W: case for 'http://www.oft.gov.uk/OFTwork/competition-act-and-cartels/criminal-cartels-completed/marine-hose' not in index
rgarner commented 9 years ago

Largely fixed by cf6f4fad8a2bb5aa827803e51ca8008e7969222d

adammaddison commented 9 years ago

\o/