CRS activities

[KitWallace]

Having loaded all those CRS activities from the unitedstates, I saw this news item

http://www.oecd.org/dac/aid-architecture/crs-xml.htm

which intimates that another 740Mb of CRS records turned transaction by transaction into IATI activities may be on the way.

Whilst I'm pleased that the database has stood up to this influx, I doubt the value of these records. They seem disassociated, both from a temporal context, from related activities and from recipients.

[KitWallace]

Following discussion with Bill, position is that flat CRS-derived activities will not be loaded into the AIDVIEW-db until they are reformatted to represent true activities.

Some consequences:

• Until this is flagged in the Registry, we cant do an automated download which excludes is data
• all unitedstates published data is to be removed
• need to be able to flag a publisher as not suitable for upload. This will require an additional dataset to represent publishers directly, since at present they are only a field in an activitySet

[KitWallace]

As a simple expedient, there is now a blacklist.xml file per corpus in {corpus}/sets omit elements define a pattern to match against the package name and if matched the text of the element is returned where it will be held as an error message in the omitted activitySet.