Parsing error on IATI sectors

[matmaxgeds]

See screenshot - somehow a project description has been parsed as a sector and so been imported as one.....

[raashidahmad]

@matmaxgeds This has been parsed correctly. The IATI Identifier for this activity is: GB-CHC-1001813-RNGSomalia. Title for this activity is: 3. Mine Action and Cluster Munitions Programme 2016 -2020

Please see some snapshots below for help:

![Uploading Long-Sector-Extraction-by-Parser.png…]() ![Uploading Project-Desc-For-Long-Sector.png…]()

[raashidahmad]

Closing it as parsing is okay, it is the data entered long for sector.

[matmaxgeds]

Thanks - I have found the source file and see that it is a problem in the IATI data. It also highlights the problem that IATI publishers are entering different narratives for the same sector code so we are ending up with many more options than we should have. We have two options, either

1. We use the IATI codelists API or another source and get the sector codes and names from there, and only go to the narrative element if the code is missing from the codelist.
2. We scrape first a list of the codes, and then to take the most common narrative that corresponds to them.

In both options, we should probably list them as 11100 Education ie. the code, and then the name in english, not the description, but the name of the 5 digit code.

I think option 1 is probably quickest at this stage. And the easiest source is here: https://datahub.io/core/dac-and-crs-code-lists/r/14.csv - or alternatively here: https://webfs.oecd.org/crs-iati-xml/Lookup/DAC-CRS-CODES.xml but that one needs a bit more parsing, or here is another source: https://github.com/IATI/IATI-Codelists-NonEmbedded/blob/master/xml/Sector.xml, or here is another source, but it is about to change so the domain would need to be editable: https://test-datastore.iatistandard.org/api/codelists/Sector/?format=json

Here is a good one for the SDG Goals: https://github.com/IATI/IATI-Codelists-NonEmbedded/blob/master/xml/UNSDG-Goals.xml and the targets: https://github.com/IATI/IATI-Codelists-NonEmbedded/blob/master/xml/UNSDG-Targets.xml

There will need to be a management page for this.

Whatever we do for the OECD DAC codes, we should do for all the other codes.

[matmaxgeds]

@raashidahmad as we have some extra time, we really need to solve this before the launch:

• [ ] Based on the codelists given in the IATI data, make a list of codelists needed. Then check if they can be downloaded from one of the sources below, if yes, then use that source. If not, then get the data as currently by using the narrative elements. This means that e.g. the OECD DAC sector codelists will come from a source, not the narrative elements.
• [ ] Import IATI sector codelists from either here: https://codelists.codeforiati.org/Sector/ i.e. see the link in the top right to get to : https://codelists.codeforiati.org/api/csv/en/Sector.csv or https://codelists.codeforiati.org/api/csv/en/Sector.json or https://codelists.codeforiati.org/api/csv/en/Sector.xml or from the source here: https://github.com/IATI/IATI-Codelists-NonEmbedded/blob/master/xml/Sector.xml
• [ ] Please add a field to the IATI management page to set the base url for use when searching for codelists
• [ ] Please import the codelist entries with the code embedded into the front of the name (this will save the ordering issue) e.g. "11100: Basic Education" also as users will often search for these codes to find the sector they are looking for and so this will work via the sector text search

[matmaxgeds]

closing as now duplicated by #163