Given a domain name, we could have a process such that:
edit.tosdr.org calls a webhook on a Heroku-hosted instance of https://github.com/ambanum/CGUs (our new Node.js-based evolution of tosback)
edit.tosdr.org and the crawler share the same postgres db instance
it will find links on the home page (footer) and ask the reviewer to pick the right link, by anchor text.
it will try to group service domains into services by looking at domain name similarity (e.g. only extension is different, like in google.*), and also if two services have the same set of documents.
In case of doubt, we do create one service per domain, but there needs to be at least one document at the service's own domain.
Given a domain name, we could have a process such that: