Import IATI sectors

[matmaxgeds]

The IATI parser doesn't seem to be picking up the IATI sectors

See for example this IATI activity, it has a sector type available that doesn't get picked up in the UI: http://datastore.iatistandard.org/api/1/access/activity.xml?iati-identifier=XM-DAC-41301-616731

- this needs to be referenced against the AIMS sector list for the vocabulary="1" to get the proper name, and presented to the user for import.

[matmaxgeds]

The current approach to importing IATI sectors is not working, the data quality is too bad so there are too many mis-spelled sectors. Instead, we need to be pulling the IATI sectors from the IATI XML files. We can do this by having a section on the IATI management page where the management user can add link to wherever they might be hosted. These are not via an API, for example: https://codelists.codeforiati.org/api/xml/Sector.xml and the management user should be able to add more links.

[raashidahmad]

@matmaxgeds The IATI sectors are imported from the IATI returned by stream=true querystring. The IATI must returned the sector names from the same source. We can discuss it in tomorrow's meeting.

[matmaxgeds]

Options and issues:

1. Replacing the sector codes with corrected ones - we use the raw xml data file, this downloads every night, therefore the sector cleaning/replacement process would also need to run every night, and the file would be unavailable then. Presumably it could run on a copy, and replace the active one when done? The management user would need to add the name of the XML field when adding the xml file for each set of sector codes so that the replacement script knew when to run
2. We could get a version of the raw data with this (the sector codes) fixed maybe, it would need to be identical to the current raw data file

See also previous discussion here: https://github.com/somaliaims/Somali-AIMSUI/issues/293

[raashidahmad]

Implemented and presented for review.

[matmaxgeds]

@raashidahmad - unclear how this has been implemented as there is still a sector called: " Support to trade and business associations, chambers of commerce; legal and regulatory reform aimed at improving business and investment climate; private sector institution capacity building and advice; trade information; public-private sector networking including trade fairs; e‑commerce. Where sector cannot be specified: general support to private sector enterprises (in particular, use code 32130 for enterprises in the industrial sector)." which is wrong, see the solution for where to get sector codes from here: https://github.com/somaliaims/Somali-AIMSUI/issues/21#issuecomment-576541088

[raashidahmad]

Implemented, presented for review in the current build.

[matmaxgeds]

@raashidahmad - that looks like it is working great. one small thing:

1. Please can you import the IATI sectors as 'code: narrative' e.g. there is a DAC code with the narrative "Primary education", and it has the IATI code "11220" - please can we store it as "11220: Primary Education"?

Codelist extract for example above

11220 Primary education Enseignement primaire Formal and non-formal primary education for children; all elementary and first cycle systematic instruction; provision of learning materials. Enseignement primaire formel et non formel pour les enfants ; enseignement élémentaire général ; fournitures scolaires. 112

[raashidahmad]

@matmaxgeds The sectors shown in the following sections now the code: sector name.

1. Data entry form: Sectors & Locations
2. Sectors mapping under Management. These are the places where the user normally deals with IATI Types.

[matmaxgeds]

@raashidahmad this looks great - thanks.

This one is not parsing somehow: https://codelists.codeforiati.org/api/xml/UNSDG-Targets.xml I don't see it in the dropdowns?

[raashidahmad]

@matmaxgeds this is because scripts only runs every midnight. Let me try to run it now so that you can see it.

[raashidahmad]

@matmaxgeds Please check it now, and close if you find it okay.

[matmaxgeds]

@raashidahmad I just looked for it here: http://aims.mop.gov.so/sector-mappings and there still was nothing in the dropdown for that one (the Targets one)?

[raashidahmad]

@matmaxgeds Hmmm, were there any sectors available before? or confirm me one more thing, how should this script behave? If there are no sectors under a sector type, because it only replaces names with the one from source, if no sectors are present, it will not do anything.

[matmaxgeds]

@raashidahmad - I think there is some confusion here - the script should extract the sectors from the sector xml file and add them to the table, it cannot wait for any replacement as otherwise there is never any way that someone can use a new sector that hasn't been used before. After it has added all the sectors to the sector table, then it can do the replacements in the activities xml file. Might that be the issue?

[raashidahmad]

@matmaxgeds Ok, i will look into the script and do the fixes.

[raashidahmad]

@matmaxgeds suggested changes implemented. Sectors are added from the provided sources. Please check the sectors and IATI Codes imported.