Improving organisation identification

[timgdavies]

We need to be able to consistently identify organisations so that It is possible:

• to identify how many organisations are active in a given location
• to identify which other donors have worked with an organisation (As a proxy for trustworthy-ness etc.)

Often organisation names are just available as free-text. This creates a challenge when:

• The same organisation name is typed differently
• Acronyms and other short names are sometimes used
• Different language versions of the same organisation name are used
• There is ambiguity over different national and local chapters of an organisation (E.g. Save International, Save Tanzania)

This requires tooling to support a combination of:

• Lookup of existing known organisations and IDs
• Disambiguation of organisations (perhaps by other known information, such as their last known activity etc.)
• De-duplication

This might be able to draw upon and improve the existing Organisation Identifier Registration Agency codelist to help select preferred identifiers.

[timgdavies]

@danmihaila we're just starting to explore this issue for OpenAgFunding project - and wondering if any of the work you have done in the past on Org IDs is still live anywhere / something you are up for developing further?

[stevieflow]

There's a related issue around this in terms of how the Organisation Registry codelist is updated within the IATI Standard - http://discuss.iatistandard.org/t/adding-to-the-organisation-registration-agency-code-list-push-rather-than-pull/414

[danmihaila]

@timgdavies we have the work on a server and I could make it available. lets have a chat before, to see what we need to do.

[timgdavies]

Dan has shared a great presentation of work mapped out last year which steps through many of the issues and describes a prototype which @danmihaila has kindly put live at http://happy-devs.atman.ro/solr/org_collection/browse to show how data aggregated from existing IATI data could be used by:

• Publishers - to select better IDs to use
• Quality assurance - to identify IDs being mis-used, or publishers who need to improve their identifiers

[dwalker101]

Also was thinking that perhaps we could implement a small string matching algorithm that measures differences in title strings -- e.g. "World Bank" vs. "The World Bank." We could flag strings that are very similar and maybe even automatically equate strings that are above a certain threshold of similarity.

[danmihaila]

@dwalker101 you are right. We tried to do something similar. If you go to http://happy-devs.atman.ro/solr/org_collection/browse and you click on "More organizations with same code" or "More organizations with same name" you will arrive here: http://happy-devs.atman.ro/solr/org_collection/browse?q=&fq=code:%22GB-1%22&sort=count%20desc&show_chart=code or http://happy-devs.atman.ro/solr/org_collection/browse?q=&fq=name:%22Department%20for%20International%20Development%22&sort=count%20desc&show_chart=name