search

kcigeospatial / MDOT-SHA-NPDES-Next-Gen

Code and issues related to the MDOT SHA NPDES Project. Project codes: Config = 31, Management = 32.
0 stars 0 forks source link

Duplicated Metadata IDs in NPDES Inventory Feature Service #360

Closed KCI-Ablowers closed 5 years ago

KCI-Ablowers commented 5 years ago

Issue

There appear to be multiple sets of metadata (KCI-1 in this case) which are duplicated within the domain on the Metadata_ID field in the Inventory Feature Service.

Background

While working on ETL Issues, I noticed that some of the Metadata Id records appear a number of times in the domain. Some appear a couple of times, while like in the screenshot below, there is a large number for April 11th and 12th 2019.

image

johnshiu commented 5 years ago

@KCI-Ablowers, it looks like there are tons of duplicates in the FLD_METADATA_INFO table for the KCI_1 version: image

This isn't the case with RJM: image

But it is the case with EAE: image

Perhaps there is something with the way the metadata was committed? IIRC, as it's a separate table, it is synced via ArcMap and not through Collector/AGOL.

Ultimately, if each date is supposed to be merged down to a single record, then I'd recommend not worrying about it too much right now and fixing it after it commits to QC/merges to source. You should just make sure to have the metadata domain associate with one of the correct dates.

After the commit to source, we can:

  1. Keep one metadata record per CREATOR/DATECREATE entry.
  2. Remap the duplicates to that one record.
  3. Delete the duplicate metadata records.
KCI-Ablowers commented 5 years ago

@johnshiu It could be the way we are committing the new metadata record, but everyone should have the same workflow. You are correct, the bring the table down locally in ArcMap, add the records we need then sync the online copy. Maybe there is an issue there, I remember us having issues with duplicate geometry the thought came from the offline copy.

I think your your solution is good and it's not a major issue. Just wanted to get something up here to track.

johnshiu commented 5 years ago

These duplicates (anything since January 1, 2019) have been removed from NPDES source and the corresponding META_IDs in SWMFAC, STRUCTURES, and CONVEYANCE have been updated.

There are also metadata duplicates in older parts of the NPDES data that I could also clean up, but I will defer that until later if needed as I don't want to mess with any legacy data.