kcigeospatial / balt_co_ETL

This will provide a place to track internal issues around the Baltimore County NPDES NCT application ETL.
0 stars 0 forks source link

Stormwater - BMP Inspections - Selection Set 1C ghosted BMP inspections are being migrated #68

Closed leeensminger closed 6 years ago

leeensminger commented 6 years ago

This is discussed in the first comment of #67 : _Also - target BMPInspections are related to BMPs. And the BMPInspections table includes inspections for these Archived records - they have an BMPID, and interestingly, because the BMP table records are not present, these inspections are orphaned. The inspections should not be loaded to target either.

Archived records are being inadvertently migrated to BMPPOI, but shouldn't be, based on SW_TRACKING.TRACKING_STATUS. The inspections related to these facilities are also being loaded to BMPInspections, and shouldn't be.

gerrykelly commented 6 years ago

@leeensminger I thought other bug fixes would have resolved this, but I get 3 BMPInspections inspections records when I use RY 2010. I'm going to ignore that for now and circle back and try to figure it out later if lt's a real problem and matters.

leeensminger commented 6 years ago

@gerrykelly I think its an issue, yes - running RY 2014 again (selection set 1A) gives me all the correct inspections for 1A, plus 9 inspections related to 1C.

These are the 9 bad records that were loaded, and shouldn't have been: image

gerrykelly commented 6 years ago

@leeensminger I never seem to get quite the same results as you, but I see the problem of the orphaned inspections. Rather than spending even more time going around in circles to find the cause, I just put in a last-step routine that deletes the orphans. I thought there was still a glitch with having more inspection records than related BMPs, but from a data perspective it looks legit with BMPs sometimes having more than one related inspection on the latest inspection date: image

leeensminger commented 6 years ago

Resolved