Manually spot check tmp collections for false positives.

[prabhushrikant]

• Whenever new data is crawled conflate it against existing data in master collection(services collection) if it already exists.
• details of conflation:
  • Fuzzy matching on name (90% or more match) (higher weight)
  • City match (match exactly)
  • State (match exactly)
  • Zip code (match exactly) (lower weight)
  • Possible match should also belong to the same category of resources.

[prabhushrikant]

Following checks can be done:

1. If the resource is providing more than one service is it duplicated in the csv data? Should we then also match on category while dupe checking?
2. Abbreviations (St. vs Saint etc)
3. Spelling mistakes
4. Spit out the list of dropped services for manual checking.

[prabhushrikant]

We have now collected a lot of tmp data collection but we would like to improve our partial matching logic. Spot check the data collected in tmp collections like tmpIRS or tmpHUD and see if those resources are already found in other collections such as services or tmpILMS

Also check if tmp duplicates are actaully duplicates found in some collection and not false positives