Entity Resolution

[wwqrd]

Include PSL-based entity resolution as an on-export feature.

• Entity resolution is performed by a java program, built using psl (https://psl.linqs.org)
• On export users should have the option to attempt to merge networks:
  • Networks would be run through the java app
  • User would then have option to combine nodes one by one in prompts

Questions

• [x] ~What does the entity resolver program look like in it's current implementation?~ Uses command line interfaces, implementation is made up of .psl (rules dsl), and a .data (defines data sources), and the data files, output will be another text file.
• [x] What are the different options for integration?

[wwqrd]

Preliminary thoughts on integration options:

Standalone

Binary would exist elsewhere and user would configure Server to point at the binary.

• Server would need to be able to output suitable input for psl
• Server would need to be able to read result
• In theory could work with anything that outputs the same format
• Low-touch option

Bundled

psl jar would be included with server

• what about java needing to be installed?
• Server would still need same input/output options as above, but would handle the files automatically.

As a specialised app

A custom app incorporating the psl library, communicating with server via a messaging service.

• use groovy/java interface to create specific jar (essentially built-in .psl/.data options)
• interface would be custom, options include:
  • messaging service (e.g rabbitmq)
  • node.js / java bridge
• Could be bundled or standalone
• Most complex option?

[wwqrd]

We're going with the Standalone option:

Standalone

• Binary is managed by the user
• Server is configured to run the binary as a separate process.
  • Input could either be a file or stdin
  • Input will be the latest transformed network + any new data since
  • Output could either be a file or stdout
• Server will need to be able to prompt for user feedback in response to output (confidence intervals)
• Server will need to store:
  • Final transformations for each run of the script.

[wwqrd]

This is from beginning of feb, but putting it here for reference:

Build plan

1. new endpoint 1: /resolve_request Will mirror export options Returns raw resolution list, and potentially original network / affected nodes
2. Basic UI that reads resolution list + original network / affected nodes
3. New endpoint 2 /save_resolution Save finialised resolutions to a new nedb Returns success and potentially triggers export?
4. Update export endpoit to optionally incorporate resolutions.
5. Style UI

Spec notes

• Node resolutions will be recursive, if a node pair is resolved earlier in the list, the next match will be shown the earlier resolution.
• Node resolutions will be given new ids
• Roster nodes (e.g. those which are built into the protocol) should not be resolved - the onus is on the external script to manage this expectation.

[wwqrd]

This issue has been superseded by: https://github.com/complexdatacollective/Server/issues/292