Closed robgur closed 12 years ago
Totally. Assigning issue to myself and changed the milestone to Demo.
Update: I'm running a MapReduce job over MasterSearchIndex and so far discovered over 1000 duplicates. Also finding entities with a parent that references an OccurrenceSet not in the datastore. I'll know more after the job is complete.
@tucotuco: As a sanity check can you see if the bulkloader is checking for existing MSI before putting new ones? That might be part of what's happening here and should be a fairly painless fix. Maybe?
Changed issue labels to Priority-High and Status-Started.
343,075 MasterSearchIndex entities, 1189 are duplicates, 27158 have non-existent OccurrenceSet parents. Here's the full list:
https://gist.github.com/1116099
Will update when I know more.
Couple options here. We can write code to fix duplicates and parent key errors, or harden the bulkloading code/process and clear/reload datastore where needed. I think the latter is a better solution, especially as we ramp up adding new layers to MoL.
I have found a couple cases of duplicate (or two separate sets of) ecoregions for a single taxon. An example is a search on "Melanoptila glabrirostris".