Open Robinlovelace opened 4 years ago
You can find a list of planning applications from PlanIt having app_size = large, app_state = permitted and end_date = 2002-01-01 under acton/releases/large_old_apps.csv.
@aspeakman I realise most of these applications are not from planning authorities that you have re-scraped data from. But what impact does that have on the app_size classifications? The list contains various applications that don't seem very large at first glance, such as "Extend and raise the roof to create a large dormer"; "Certificate of Lawfulness for the proposed replacement of windows"; "Consultation by Cornwall County Council for the construction of extension to kitchen.", etc. How would the re-scraping change this?
Great progress, simpler than initially thought it seems, and sounds like we have enough to get going on creating a model and writing-up as a plan of action (or should I say a plan of acton heheh ; )
Sorry but many of these applications were misclassified during my first pass at this when I thought that keywords 'prior approval' or 'prior notification' in the application_type field signified a large application. Now the decision is mainly based on the n_statutory_days and n_documents fields (where available).
For around 60% of authorities having a particular type (Idox) these fields are being collected and updated BUT applications before 2005 have not yet been included, because this is a cut-off date for arbitrary processing time reasons (especially noticeable in Cornwall). I hope to target these pre-2005 applications for update at some point .
Note across the board there is a long tail of other scraper types that each require an update to the source code to collect a value in the 'n_documents' field which is one of the key identifiers for a large application. In the mean time these other types will make the 'app_size' classification based on the 'n_statutory_days' field (>60)
Thanks for the updated @aspeakman.
I hope to target these pre-2005 applications for update at some point .
Is there any way you can fast track this? The project ends soon and getting a good handle on pre 2011 developments will be most useful in the next 24 hours if at all possible.
For various reasons there are quite major lags built in to the system. There are around 265 Idox scrapers and they are mostly quite slow. In 2016 I was scraping a few of the early examples back to year 2000 (hence the old data in that range), but as the number expanded I had to restrict the current data scraping to 2006 onwards. I recently moved the target scraping date back by a year to 2005 and it took more than a month for the system to catch up across the board.
So I can offer some targeted updates (but not within 24 hours). Alternatively you could restrict your analysis to 2005 to 2011 to provide better results. Or I can provide the list of target dates for each scraper so you know how far back you can trust the data in each case.
Thanks the quick reply @aspeakman, will defer to @joeytalbot on best option.
Note that the PlanIt API now has a new 'Conditions' application type, which has been split out from Amendment.
You can see this in action at: https://www.streetfocus.uk/map/
Another potential source of data for future developments:
Related to this: #56