[WIP] Reconciliation Service

opencivicdata / docs.opencivicdata.org

Open Civic Data project documentation

https://open-civic-data.readthedocs.io

44 stars 33 forks source link

[WIP] Reconciliation Service #92

Open fgregg opened 7 years ago

fgregg commented 7 years ago

This OCDEP proposes a design spec and governance model for an opencivicdata entity resolution service. This will allow publishers of civic data use the same ocd ids while staying loosely coupled.

fgregg commented 6 years ago

Large outstanding question: How to handle existing ids

I've been reflecting on Julie A McMurry, et.al.'s excellent paper on identifiers, where she advises "only creating your own identifiers for new knowledge" While this sounds sound, I'm not sure how to square this with our requirement that our identifiers be ocd identifiers.

If we do try to heed the advise that we not create new, redundant identifiers, then who will be responsible for synchronizing with these other authorities?

If that's too much of a dependency, should we allow for search by external ids?

fgregg commented 6 years ago

Some notes

[ ] should we require some sort of proof that a search was made before creating new id? Something like this his would be necessary for training, maybe
[ ] if we allow a delete record method we need to decide what happens when last record was deleted
[ ] should we have a source field in records
[ ] we need to encourage putting after matching
[ ] should the search method return anything beyond the ids, maybe the nearest matching record?
[ ] should have a "what this is not section" i.e. this is not an attempt to integrate information from various datasets or provide canonical information.

fgregg commented 6 years ago

This is ready for a preliminary review. There's a number of governance and incentive questions that I still have, but I figure those can wait until we have worked on some implementation.