Reconciliation / de-duplication

[timgdavies]

We need to handle the following case:

• A user has a spreadsheet with project names, company names and license names, but no identifiers for them
• Some of these entities are already known about by ResourceProjects.org
• The identity of some of these entities might be ambiguous
• They name in the users spreadsheet might not exactly match the name in ResourceProjects.org

Potential solutions:

• During the process of preparing data, we want to reconcile the names to identifiers from ResourceProjects.org
• During the conversion process, we reconcile these names to identifiers from ResourceProjects.org
• During the import process ....

Challenges:

• User input to agree to the matches suggested might be required;
• There may be a gap between reconciliation when a user fetches IDs, and the time they upload their data, during which new IDs might become available in the system.

Things to explore:

• Refine reconciliation API / services: https://github.com/OpenRefine/OpenRefine/wiki/Reconciliation-Service-API
• LOD2 Stack components: http://stack.lod2.eu/blog/

We need to:

• Develop the technical components required;
• Document the workflow to use them.

[timgdavies]

I've been finding a lot of my work to prepare data is taking place in Open Refine - and so we might want to focus on developing reconciliation processes here in the first instance.

[timgdavies]

The reconciliation approach is now primarily taking place through Google Spreadsheets tools.