search

WPRDC / wprdc-etl

MIT License
8 stars 3 forks source link

PLI Data problems #30

Closed bsmithgall closed 8 years ago

bsmithgall commented 8 years ago

Looks like there are some problems with the PLI data -- I have missing inspection dates and next action dates in 20160.

Should this be fixed at the source? Should the data be dropped if it hits a validation error? What is the right way forward for you?

bsmithgall commented 8 years ago

Note: this blocks #2 a bit.

saylorsd commented 8 years ago

I just downloaded a recent copy of their pli data, and none of those issues are there anymore. The 20160 thing seems really weird, like a \r\n is missing after the 2016. I'm hoping that was just some weirdness with me sampling it.

Regarding the missing inspection date, this may be something that does occur occasionally and then is fixed within a day or two. I think these should be considered NULL, and when we use the Datastore's upsert call, these will be updated.

I'll make a PR with some new test data.

bsmithgall commented 8 years ago

This was fixed in a PR right? If so, can you close it?