search

adrianshort / uk_planning_scraper

A Ruby gem to get planning applications data from UK council websites.
GNU Lesser General Public License v3.0
27 stars 19 forks source link

Ensuring that you have **every** planning authority in the UK #17

Open jnicho02 opened 6 years ago

jnicho02 commented 6 years ago

I've started adding some of the authorities that I found on https://www.local.gov.uk/our-support/guidance-and-resources/communications-support/digital-councils/social-media/go-further/a-z-councils-online and found a special one referred to by the Adur web site...South Downs National Park. Which makes me wonder how/where to get a full defacto list of planning authorities. It may exist somewhere of data.gov

adrianshort commented 6 years ago

I've asked MHCLG and apparently there isn't a definitive register of UK planning authorities. There are registers of local authorities in England, Wales, Scotland and Northern Ireland. And then you have to fill in somehow (?) with the national parks, development corporations, GLA, etc. to get every body that's a planning authority.

See the developmentcorporations tag. We're going to need a nationalparks one too.

adrianshort commented 6 years ago

I haven't tried it yet but I assume we could find many of the search URLs for many authorities by spidering council websites from their homepages. Look for the planning section, then links with titles like planning register, search, find or comment. And in many cases we can recognise the target using the Idox/Northgate etc regexen that are already in Authority#system.

adrianshort commented 6 years ago

Another tell for the target URL would be that it's on a different subdomain to the main council website. You can see the common subdomains by parsing the URLs we've already got.

jnicho02 commented 6 years ago

I'll ask around. Am going to OpenData Camp in Aberdeen next month and people there might be able to help.

adrianshort commented 6 years ago

Thanks. The other issue is authority IDs: the local authorities have got GSS IDs but the non-local authority planning authorities don't.

jnicho02 commented 6 years ago

So, gov.uk registers....

These could be accessed via API probably pretty safely, or downloaded. They also provide a unique ref for each authority which would be authoritative.

adrianshort commented 6 years ago

Using the API is fine in terms of staying up to date but then we can't manually add tags.

We could use something like MapIt to do all the geo stuff automatically but that would require some thought.

dhilton commented 6 years ago

Some LA planning portals: https://gist.github.com/dhilton/4d714c38cb1ae5acf73ac7b3af242d10

adrianshort commented 6 years ago

Thanks. That's very useful. How did you find them?

dhilton commented 6 years ago

https://www.local.gov.uk/our-support/guidance-and-resources/communications-support/digital-councils/social-media/go-further/a-z-councils-online + subdomain bruteforcing

adrianshort commented 6 years ago

@aspeakman has got a comprehensive set of UK Planning scrapers in Python, although I'm not sure how up to date they are. Here's Andrew's list:

https://github.com/aspeakman/UKPlanning/blob/master/scraper_list.csv

aspeakman commented 6 years ago

You can access details of all UK planning authorities on my website planit.org.uk. There are 432 planning authorities linked together with relevant geographical and administrative entities.

See http://www.planit.org.uk/find/areas/

The full data can be downloaded as JSON via http://www.planit.org.uk/api/areas/json?pg_sz=512

adrianshort commented 6 years ago

Thanks @aspeakman, that's extremely helpful. And a great project.