[IDEA] Maintain replicated codelists in IATI format

[andylolz]

Rationale

The IATI standard includes a number of replicated codelists, that come from external sources (e.g. DAC; IANA; ISO). These are currently maintained by the secretariat, but the update process is entirely manual, so there’s a lot of overhead. As a result, it’s sometimes difficult for codelists to be kept up-to-date.

Proposal

Maintain a set of these replicated codelists in IATI format. Auto-update them using scrapers, in much the same way as the EveryPolitician bot. There are currently 22 replicated codelists, so this is a very manageable number. However, I know there is interest from @matmaxgeds in expanding this to other codelists that are useful to publishers but are not currently part of the IATI standard.

The replicated codelists could be outputted in XML, JSON, CSV and nicely formatted HTML.

---

The name Tyrell is a reference to the replicant manufacturing “Tyrell Corporation”, in the 1982 neo-noir sci-fi film Blade Runner. We think this is probably a bit sinister! Need a better name :)

[andylolz]

I’ve marked this as "in progress" because I’ve already created scrapers for several of the codelists in question.

[matmaxgeds]

I am guessing that once we have the scrapers outputting to a repo, we can have a job/tool that calls csvdiff, xmldiff etc to monitor when the scraped and the IATI version differ - and alert us - bonus points if this was output to a file that we could suck into a one page traffic light dashboard - even more bonus points if it was automatically tweeted ala IATI canary but for codelists.

[markbrough]

A few ideas:

• multilingual - endeavour to make codelists available at least in EN and FR, preferably with some way of gradually expanding to cover other languages over time
• include other helpful information e.g. whether a code has been withdrawn, whether a code is recommended to be used (e.g. along lines of budget identifier guidance); mapping files to equivalent / recommended new codes when a code is withdrawn
• also include (access to?) other useful codelists, e.g. organisation identifiers; exchange rates

[andylolz]

Ace – great stuff both.

I’ve made a new repo here: https://github.com/codeforIATI/codelists

I’ll transfer those ideas across to issues in that repo now shortly.

[andylolz]

@markbrough has done some more on this one :tada:

[andylolz]

I’ve moved suggestions to https://github.com/codeforIATI/codelists/issues.

Please do keep adding to that list! It would be good to get a list of features from which to assemble a bit of a spec.

[andylolz]

This exists! https://codelists.codeforiati.org

There’s more work to do on this, but closing it here since we have a dedicated issues board for it.