search

adrianshort / uk_planning_scraper

A Ruby gem to get planning applications data from UK council websites.
GNU Lesser General Public License v3.0
27 stars 19 forks source link

Raise exception when Idox search fails due to too many results #14

Closed adrianshort closed 5 years ago

adrianshort commented 5 years ago

Currently this fails silently with an empty array returned from Authority#scrape, eg:

UKPlanningScraper::Authority.named('Newham').scrape({ validated_days: 90 })
 # => []

UKPlanningScraper::Authority.named('Newham').scrape({ validated_days: 9 })
Using Idox scraper.
Getting: https://pa.newham.gov.uk/online-applications/search.do?action=advanced
Found 10 apps on this page.
...

The search form returns this error in HTML:

<div class="messagebox errors">
  <h2>Please check the search criteria:</h2>
  <ul>
    <li>Too many results found. Please enter some more parameters.</li>
  </ul>
</div>