pudo
commented
6 years ago Thanks for starting the issue. Let's move on this. I think it's mainly an issue of defining a nice set of interfaces that we can work with, and then filling them with functionality for the individual enrichment APIs.
Our goal should be to expose this functionality both as a gRPC for Aleph, and as a command-line utility (which would likely run either against local CSV files, or simple backend like SQLite) for use in investigations like the Laundromat.
Let's think about the API for this, we can even do it in a test-driven way where we have a fake backend and a round of tests for it. Here's a little proposal:
a) We expect an entity to be a dict. Inside the dict is an id (which is a sha1 of something that uniquely identifies that entity), a schema (which defines the FtM type of the entity) and a dict of properties. Each entry in the properties is a valid FtM model property (cf. followthemoney/schema/*.yaml). It's value is a list of values.
b) Using this definition of an entity, we can define some API classes:
class EnricherResult(object):
def __init__(self, score, entities):
self.score = score
self.entities = entities
class Enricher(object):
def enrich_entity(self, entity):
raise NotImplemented()
def enrich_query(self, query_string):
raise NotImplemented()
class OpenCorporatesEnricher(Enricher):
def enrich_entity(self, entity):
# query open corporates
if (entity['schema'] in ['Person', 'Company', 'LegalEntity', '...']):
# do a directors' search
for result in self.enrich_director(entity):
yield result
if (entity['schema'] in ['Company', 'LegalEntity']):
# do a company search
for result in self.enrich_company(entity):
yield result
def enrich_company(self, entity):
# make a query from entity['properties']['name'], entity['jurisdictions'] etc.
...
for res in response['companies']:
result = EnricherResult(res['score'], [])
ftm_company = self.align_oc_company(result)
result.entities.append(ftm_company)
for director in res['directors']:
ftm_director = self.align_oc_director(director)
result.entities.append(ftm_director)
ftm_directorship = self.align_oc_directorship(director)
result.entities.append(ftm_directorship)
yield result
Does that make sense as an API?
jcshea
Build a revised version of corpint data enrichment that will be designed to automagically build company profiles for laundromat companies (as a first target). Features including :