adrianshort / uk_planning_scraper

A Ruby gem to get planning applications data from UK council websites.
GNU Lesser General Public License v3.0
27 stars 19 forks source link

Idox eg Newham: Some authorities don't have received from/to dates on search form #6

Closed adrianshort closed 5 years ago

adrianshort commented 5 years ago

Can we just check if the fields are present and add them if they're not?

UKPlanningScraper::Authority.named('Newham').scrape({ received_days: 1 })`

raises:

NoMethodError: undefined method `date(applicationReceivedStart)'
adrianshort commented 5 years ago
# Print the authority names that don't have received dates fields on their search form
require 'uk_planning_scraper'

auths = UKPlanningScraper::Authority.tagged('idox')
agent = Mechanize.new

auths.each do |auth|
  begin
    page = agent.get(auth.url)
    next unless form = page.form('searchCriteriaForm')
    puts auth.name unless form.has_field?('date(applicationReceivedStart)')
  rescue StandardError
  end
end

returns:

Hammersmith and Fulham
Newham
Manchester
Stockport
Bristol
Leeds
Sheffield
Bradford