proposal: v2 Search endpoint and generalized entity model

With the recent additions of entity lists from around the world we've started to notice there are many overlapping fields (often named slightly differently) and the current /search endpoint has grown well beyond its initial scope. The current endpoint returns many key/array-of-objects results which are hard to parse though.

This proposal is two fold.

Create a generalized model for entities
Create a /v2/search endpoint which accepts generalized field queries to search across all lists

Generalized model for enties

The idea of a generalized model would be to offer a shared high-level search across all supported lists and shared fields for every entity supported by Watchman.

Initially the modeling could look like the following:

type Entity[T any] struct { // TODO: list all of the shared fields
    Name string `json:"name"`
    Type EntityType `json:"entityType"`
    Source SourceList `json:"sourceList"`

    Addresses []Address `json:"addresses"` // TODO: define 

    SourceData T `json:"sourceData"` // Contains all original list data with source list naming
}

type EntityType string
var (
    EntityPerson EntityType = "person"
    EntityBusiness EntityType = "business"
    EntityAircraft EntityType = "aircraft"
    EntityVessel EntityType = "vessel"
)

type SourceList string
var (
    SourceOFAC SourceList = "ofac"
    SourceUSCSL SourceList = "us_csl"
    SourceUKCSL SourceList = "uk_csl"
)

Q: Should we adopt the OpenSanctions modeling?

v2 Search Endpoint

Create a new /v2/search endpoint which accepts all of the generalized entity fields as parameters and searches across all supported lists. Results are returned in their generalized form with a sub-object of their source list data. (In the original naming of the source list.)

GET /v2/search?name=...&country=....&type=individual

{
  "entities": [
    {
      "name": "John Doe",
      "entityType": "person",
      "sourceList": "us_csl",
      "addresses": [
        {
          "address1": "..",
          "country": ".."
        }
      ],
      "sourceData": {
        "name": "John Doe",
        "sdnType": "individual",
        "addresses": [
          {
            "address1": "..",
            "country": ".."
          }
        ],
      },
      "match": 0.754213444
    }
  ]
}

Pros: One stop source for search with a basic interface across all lists.

Cons: Dynamic types can be annoying to deal with in some languages.

moov-io / watchman

proposal: v2 Search endpoint and generalized entity model #527

Generalized model for enties

v2 Search Endpoint

Related Issues