ksmpdx
commented
8 years ago Yes, I feel your pain. The info coming out of the City's current permitting DB leaves much to be desired. Good news is we're moving to a new system. Bad news is that it will be 2018 until it's fully up-and-running.
In the interim, a couple of options. If you only care about permits that create new residential units, we have all of those mapped going back to 1996 (current through the end of July). Happy to pass along. I can give you a point shapefile.
If you want all permits, we have an x/y for all those that reconcile to an existing taxlot via the taxlot's State ID. This is the information you see in PortlandMaps. Unfortunately, any that don't reconcile -- because the State ID no longer exists, it was typed incorrectly (I know), etc. -- are lost. Could be as many as 20% of all permits. We are currently building out a state ID history table will be used to better match permits to taxlots. This should have a much higher success rate than the current method. Hopefully we'll use this method to map all permits, develop a workflow for keeping it current, and then put the data out there for people to use sometime in the next 3-6 months. Going to involve some manual cleanup to get to the initial dataset. Working on demolitions first, as there's a lot of interest and not much info available.
Bottom line, address is a dead-end with the current system. The taxlot State ID is the best way to tie permits to a known location, in this case the X/Y centroid of the lot.
Hope that helps!
caged
ungoldman
@ksmpdx Is there a way to get the
address_id, and potentially thex,yfrom the city's address data added to the building permits data? I'd like to see the geo location of the permits, but the data currently doesn't include this information. I've been trying to sync these up for a better part of a day by comparing the full addresses to each other, however I've run into multiple issues.The first challenge was a set of small issues that I bumped into before actually trying to compare the addresses.
snake_case_names." Case Number"is different than"Case Number".The next challenge in trying to sync these up manually is that there are 615,752 permits in the permit dataset and 416,424 address points. For each permit, we need to potentially scan every address to find one that matches, which is
615752 * 416424. That's a potential 256,413,910,848 iterations in a worst-case scenario.There are ways to reduce the number of comparisons that have to be made, such as caching the address for the permit
permit_case_number, since there are multiple entries for a single permit case number as the particular property has to go through multiple inspections.Now the problem moves on to comparing the actual addresses with each other. The major problem here is that it's a very finicky process and using a tool like csvjoin requires an exact match. I'm currently in this part of the process and I've hit a couple of snags pretty quickly.
There are double spaces in the full address strings. This can be difficult to spot and doesn't result in an exact match. It's easy enough to collapse spaces for the entire file. After doing this I ended up getting matches for roughly 40k permits, or about 9% of addresses and 6% of permits.
After getting the first batch of matches, I noticed that the reason more addresses aren't matching is because most of the permit data addresses end with " PORTLAND," however the address data file doesn't follow this pattern.
And, that's where I left things. I kinda felt like I went pretty far down the rabbit hole already, and maybe I should stop to see if I could get someone at the city to add this information if it's possible.
Thanks!