Add ability to mark reconciliation property as required (used for candidate retrieval)

wetneb commented 1 year ago

Currently, reconciliation queries consist in:

a name to search for (query)
a list of properties to refine the search (properties)

Each of those properties comes with a property identifier (pid) and a value (v).

This lets clients refine the search for reconciliation candidates, by providing additional data to be taken into account.

Most reconciliation pipelines feature two separate steps:

candidate retrieval: extracting a set of candidates which are likely to match the query. This is generally done using indices (also called "blocking" techniques in the literature - basically anything that lets the system avoid doing a full scan of the database but instead locate the candidates efficiently).
candidate scoring: this smaller set of candidates is processed one by one, comparing attributes of the retrieved entities with the query, to come up with a matching score

At the moment it is up to the services to determine which parts of a reconciliation query they use for candidate retrieval or matching. This is often dictated by the structure of the database they are exposing, as indices are expensive to construct and maintain.

However, in certain cases it would be helpful to give the user some control over which parts of the query should be used at the retrieval stage.

Consider for instance a batch of reconciliation queries for cities, consisting of a name (in the query field) and of geographical coordinates (in some property supplied in the properties field). Potentially, the reconciliation service could have two indices at its disposal:

a full text search index on the entity names, which could be used to retrieve candidates using the supplied names
a geographical index on the entity coordinates, which could be used to retrieve candidates using the supplied coordinates.

With their knowledge of the quality of the data to reconcile (and of the structure of the authority database), the user could either decide:

that names are more reliable, hence they should be used for candidate retrieval, and the coordinates should only be used as tie-breakers between namesakes at the scoring phase
or that coordinates are more reliable, so they should be used for retrieval, and names are then used for scoring in the second step

At the moment our API does not let the user specify this sort of information. I think it would be worth introducing a syntax for that, for instance (random JSON syntax made up without putting a lot of thought into it):

Scenario 1 would be

{
    "query": "Cambridge",
    "query_hint": "retrieval",
    "properties": [
         {
             "pid": "coordinates",
             "v": "52.205278,0.119167",
             "hint": "scoring"
         }
    ]
}

and scenario would be:

{
    "query": "Cambridge",
    "query_hint": "scoring"
    "properties": [
         {
             "pid": "coordinates",
             "v": "52.205278,0.119167",
             "hint": "retrieval"
         }
    ]
}

Because we cannot reasonably expect services to have indices on every property they expose, I think this feature should be a SHOULD or MAY and not a MUST in the specs (hence the proposed syntax of hint, that the services can decide to follow or not).

Yes, this issue is still needed in some regard, but perhaps not 100% urgent? The use case for me came up when authority ids are re-minted in the case of development with Wikibase. In certain domains/namespaces, the authority ids are valid and I want to use them. In the other domains/namespaces, I want to hint to reconcile only using the names, as the ids are suspect in the borked Wikibase db merge that happened.

reconciliation-api / specs

Add ability to mark reconciliation property as required (used for candidate retrieval) #101

See also