Entity resolution v1 reintegration

[wwqrd]

This takes the modules developed in the original development of entity resolution and re-integrates them with Server.

Entity Resolution now takes place on it's own screen, not as part of the Export Screen.

• Resolved networks can be exported from this new Entity Resolution screen with the default export settings.
• This separation of concerns is in anticipation of the export screen becoming a modal process which could interop with the entity resolution feature.

The backend could do with some more unit tests, but I think this is in a good place for beta/developer testing.

The sample repo has been heavily updated to give some guidance on how to use this feature along with an example script. https://github.com/complexdatacollective/entity-resolution-sample. A compatible example protocol is specified there: https://github.com/complexdatacollective/entity-resolution-sample/tree/master/protocol.

Feedback

• [ ] We need to add user facing text to the interface generally.
  • [x] Explain the feature and each of the items, and
  • [ ] link to the documentation
  • [x] Not sure about the numbered sections here, since this isn't really a 'flow' in the same way as export.
  • [x] Dialog when going back: "looks like you have set some attributes" - this needs to properly explain the situation to the user!
• [ ] Create a tutorial for the documentation website that outlines and demonstrates this feature
  • [x] Create a simple entity resolution protocol for the tutorial
  • [ ] Make sure to document output data format _caseID and _parentID (these may be renamed - see below)
  • [ ] Document requirements of script itself in terms of IO
  • [ ] Provide helpful troubleshooting tips for common error cases
• [ ] Improve example entity resolution script repo. This was intended to be a template repository. Align with the tutorial
  • [ ] Needs a readme
  • [ ] File organisation
  • [ ] code cleanup and comments.
• [x] BUG: "cannot resolve variables of undefined" -> when protocol has no ego stages
• [x] BUG: removing all resolutions doesn't allow the user to change the alter type for casting ego
• [ ] BUG: removing the resolution and then running another resolution shows the old number of resolved nodes in the summary
• [x] Improvements to resolution UI itself
  • [x] pressing enter should go to the next match
  • [x] high frequency responses from the resolution script still cause render thrashing and unresponsiveness. we need to throttle re-renders to every 2 seconds or so
  • [x] "use all" -> Use all attributes from this node (maybe alt text?)
  • [x] Button for matching rows shows even when number is '0'
  • [x] Dialog doesn't use available vertical height. In general show as many attributes as possible
  • [x] Improve the final summary UI by adding some explanatory text
• [x] Data format issues/errors. Some of these could have been caught by validating the output against our schema.
  • [x] session export time is the wrong format (correct format is xs:dateTime "2021-02-16T09:01:17.342Z")
  • [x] case ID and session UUID aren't correct (nc:caseId="AsiZrUAiAP1H7KlW" nc:sessionUUID="entity_resolution"). Case ID should be "Entity Resolution", and session UUID should be a uuid.
  • [x] Use the prefix "networkCanvas" rather than underscore for the caseID and parentID variables, and use better naming. So, networkCanvasOriginCaseIDs and networkCanvasOriginUUIDs.
  • [x] Ego variables and alter type variables appear separately, although ego is 'cast' to an alter type
  • [x] There is an orphan data node right after the graph element: <data key="networkCanvasUUID">undefined</data>. Suspect this is a network-exporters bug due to having no ego entity
• [x] When importing a resolved network back into server "cannot read property variables of undefined" - separate bug assuming ego again? This is not a function we want to support (see below)
• [x] Ensure resolved networks cant be imported back into Server. Perhaps we should re-establish the 'useNCGraphML' flag in network exporters, so that resolution can output 'vanilla' graphml?

[wwqrd]

There's a work in progress readme in the entity sample doc, also outlines the IO, probably needs some more elaboration: https://github.com/complexdatacollective/entity-resolution-sample/blob/master/README.md

[wwqrd]

case ID and session UUID aren't correct (nc:caseId="AsiZrUAiAP1H7KlW" nc:sessionUUID="entity_resolution"). Case ID should be "Entity Resolution", and session UUID should be a uuid.

Any thoughts what the UUID should be? Using the ID of the exported resolution currently.

[wwqrd]

Ego variables and alter type variables appear separately, although ego is 'cast' to an alter type

This is by design, probably best to have some sort of meeting to discuss.

edit: The short version is that non resolved egos aren't included in the network. We recently decided to add these "orphan" egos back in as their own node type.

[jthrilly]

Any thoughts what the UUID should be? Using the ID of the exported resolution currently.

It should be a new UUID, IMO. This is a new session conceptually.

Ego variables and alter type variables appear separately, although ego is 'cast' to an alter type

This is by design, probably best to have some sort of meeting to discuss.

Yeah, I see where you're coming from. I suppose what confuses me is that during the resolution, the user sees ego cast as a "node". So say both ego and 'person' have a 'name' attribute. Wouldn't it make sense for the user to be able to resolve what the "name" of the resolved node actually is? Right now, they can only have both the name from the 'person' node AND the name on the ego node, without merging them.

[wwqrd]

It should be a new UUID, IMO. This is a new session conceptually.

Okay I'll have it generate one when the resolution is saved, that way it can be a proper uuid, as well as being reproducible.

[wwqrd]

BUG: removing the resolution and then running another resolution shows the old number of resolved nodes in the summary

I'm not able to reproduce this one? The resolutions are listed date descending, might be that?

[wwqrd]

There is an orphan data node right after the graph element: undefined. Suspect this is a network-exporters bug due to having no ego entity

This is fixed by https://github.com/complexdatacollective/network-exporters/pull/27

[jthrilly]

Looks like the CI build error has been encountered by others: https://stackoverflow.com/questions/66331991/attempted-import-error-default-is-not-exported-from-assertthisinitialized